Scrapy is a high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages.

josefonseca · 2012-05-24T20:18:17.500854+00:00

Does anyone have any experience using this? I could see testing it out against my proprietary scripts that access the Amazon API.

harikt · 2012-05-24T18:00:23.259870+00:00

I guess I spend a lot of my time on things like conversion optimization which contributes directly to my bottom line, not worrying about IDE settings. It's probably why I make so much money :)

pemdas21 · 1 day

If anyone can toss in where a Continuous Integration server can fit into all of this that would be appreciated as well!

grotm001 · 2 days

What is a front end PHP developer?

daschl · 2 days

I'm still left with some questions:

What exactly I should do with an exception after I catch one? In his examples he just throws another exception.
When should I write a thow, catch block? When I'm expecting a possible exception? Why don't I just write all my code in one big throw, catch block?
What if I want to reuse the way I handled an exception? Can you somehow reuse the functionality that you write in a catch block?

thesplashfactor · 2 days

I agree, it's their fault, for hiring someone so grossly under-qualified/incompetent.

thesplashfactor · 2 days

I feel sorry for the php shop that hires the kid that can't edit that file.

thesplashfactor · 2 days

The thing I have come to to adhere to in appreciating jtreminio is paying special attention to the four letter tag to the right of his username

harikt · 2 days

Anyone notice this?

Property names SHOULD NOT be prefixed with a single underscore to indicate protected or private visibility.

Not a big deal but I've certainly always seen the underscore used in examples that show "best practices".

harikt · 2 days

If I do start adhering to this, does this mean I must literally count out spaces every time I start a new indented line or is there some way modern IDE's can work with this? Can you make the tab button = 4 spaces in a modern IDE for example? Or will I have to start counting out 16 spaces and stuff like that (seems like a bitch)?

harikt · 2 days

Thus, the benefit of this guide is not in the rules themselves, but in the sharing of those rules.

DreadPirateJay · 4 days

What I would absolutely be interested in is an overall summary of your workflow if you don't mind sharing. I own my own website which I want to bring other developers on board for the first time and I am working hard on developing a system that would work well for multiple devs.

Right now I'm working on getting a Jenkins CI server running with my website where builds are automatically built when someone pushes to the site's github repo. I think I will setup one development branch and one production branch and have the Jenkins CI server automatically push to the production branch if all tests pass after it makes a build. Unit testing will obviously be a big part of this process. Remote developers will be allowed to clone the development branch and then I suppose they can request my permission to merge back to it. I'm not sure tho really, I'm still working on getting this all set up and working out all the details. If you have any insight it would be greatly appreciated.

DreadPirateJay · 5 days

Does anyone here actually do TDD? do they recommend it?

beatsforthemind · 7 days

Cool, thanks for stopping by.

beatsforthemind · 7 days

This guy isn't building Amazon here dude, you sound like you're the kind of guy who enjoys over-engineering.

beatsforthemind · 7 days

It's not slow
There is no authority saying you should strictly use mysiam over innodb for instance, mysiam is inherently faster in doing sum calculations.

If you do want innodb I believe Sphinx in fact requires it and is built by the same guy (or at least he was heavily consulted by the mysql devs in developing mysql full text).

KerrickLong · 7 days

Fair enough but I'll assume them to be the authority before I do some article on css-tricks.com

KerrickLong · 7 days

If SASS is so great, why did Twitter Bootstrap go with LESS? The reality is is that I'm going to have to go through a LESS tutorial at some point to get bootstrap's full functionality.

beatsforthemind · 7 days

mysql full text indexing can be quite effective and is easy to set up.

andyana · 7 days

I just set up a Jenkins CI server on my dedicated server and it has a LDAP authentication module. I don't know what LDAP is so I was looking at it and then at a LDAP tutorial but decided it was too complicated for the time being. Can anyone explain to me what this LDAP stuff is? It has an AD module as well for linux that's built on LDAP.

If you can explain it to me briefly I'd appreciate it. Is LDAP good stuff? Is it important? Is it worth me pursuing to set up for Jenkins?

OutThisLife · 7 days

I don't think anyone's arguing with you about that.

OutThisLife · 7 days

I haven't used LESS yet but is it just me or doesn't it look much nicer to read than native CSS?

j4ck4ll · 8 days

This looks ok but where are the submit buttons and how are the forms all submitted at once? I don't see how all the information could be collected from all the forms without using javascript.

You might have to use some jQuery to collect all the info from the various forms and then supply it to the processing script.

This really isn't how html forms should work, at least not if they aren't being submitted by AJAX. Something like this would be better:

<form method="post" action="">
    <label for="data[0]">Form 1: </label>
        <input name="data[0][]" type="text" /> 
        <input name="data[0][]" type="text" />
    <label for="data[1]">Form 2: </label>
        <input name="data[1][]" type="text" /> 
        <input name="data[1][]" type="text" />
    <input type="submit" />
</form>
<?php print_r($data); ?>

try this.

magenta_placenta · 9 days

hi where do you work?

WillNye · 9 days

Tell them Mark Zuckerberg and Larry Page are web developers, then tell them to look at their bank accounts.

lulzipops

TROPHY CASE

lulzipops

TROPHY CASE

you'll need to login or register to do that

create a new account

login