Carl Burks is a software developer for a global financial institution. With over ten years experience in technology and software development for financial organizations and over twenty years of software experience, Carl Burks provides articles, musings and insight into technology issues, software development, and other selected topics.

Getting Scrapy, a web crawler framework, working on Windows 8.1

2015-01-05T20:31:00.000-08:00

Authors:
Carl Burks
First get setuptools installed to use PyPi:
https://pypi.python.org/pypi/setuptools#windows-powershell-3-or-later

This assumes you have python aliased to your instance of python in your environment:
(Invoke-WebRequest https://bootstrap.pypa.io/ez_setup.py).Content | python -

https://pip.pypa.io/en/latest/installing.html

python.exe .\get-pip.py

How can I tell that I have python setup properly in my environment?
(Get-ChildItem -Path env:* | Where-Object -Property Name -Contains -Value "Python").Length -gt 0
What does this return?
How about this:
(Get-ChildItem -Path env:Path).Value.Contains("Python")

If one doesn't return true then you might need to setup python.

https://docs.python.org/2/using/windows.html


Adding to Environment:
C:\Windows\System32\SystemPropertiesAdvanced.exe

Edit Path variable and add a semicolon and your path to Python.

Running the first command is going to fail because your system probably isn't setup to run scripts from the Internet. This is probably a good thing. You can download and save the script before executing it, or change your execution policy to allow scripts from unknown sources. That last option is a mite dangerous.

You'll need to unzip it so if you've got 7zip installed you can probably do something like this:
& 'C:\Program Files\7-Zip\7z.exe' "X" .\setuptools-11.3.zip

Keeping modules in order might require something like Virtual Environments.
pip install virtualenv

http://docs.python-guide.org/en/latest/dev/virtualenvs/

pip install scrapy

if you haven't installed it already you will need PyWin32

http://sourceforge.net/projects/pywin32/