Everything Search

I want to give an unsolicited shout-out to the one program that I recommend you download right this second:  Everything Search (link).  It’s a local file search tool which requires a few tweaks in settings (excluding %appdata%, windows, program files).  The best about this is it’s fast, not resource intensive (except for when indexing files,…

Blueprint for Sec filings Data Pipeline (ETL)

A popular source of information are the filings held in the EDGAR database. EDGAR, the Electronic Data Gathering, Analysis, and Retrieval system, performs automated collection, validation, indexing, acceptance, and forwarding of submissions by companies and others who are required by law to file forms with the U.S. Securities and Exchange Commission SEC (U.S. Securities and…

Request HTML Celery, RabbitMQ, and Redis

  To run need to do two things (while in folder of python filename.py): 1)  Run worker by executing the python program with the “worker” arguement: $ celery –A tasks worker –loglevel=info 2)  Call the task aka run: $ python filename.py ####################### # grabhtml.py import requests from html import unescape class GrabHTML(object): def __init__(self): pass…

SQL Server Create New Table And Insert Data using Python

Python Script to insert CSV File into SQL Server Database import pandas as pd import csv import pyodbc import sys, os USERNAME = ‘sa’ PASSWORD = ‘password’ SERVER = ‘server’ DATABASE = ‘DATA’ DRIVERNAME = ‘ODBC Driver 13 for SQL Server’ cnxn = pyodbc.connect(‘Driver={‘+DRIVERNAME+’};Server=’+SERVER+’;Database=’+DATABASE+’;uid=’+USERNAME+’;pwd=’+PASSWORD) cur = cnxn.cursor() ##### using pandas grab first row for column…

Working On–Week Review

Searching Website/Html/Document  – currenly using Docfetcher, PowerGrep Flask/Django – looking into creating search page for html files Docker, MongoDB (3T MongoChef), Redis (Redis Desktop Manager) in Docker Storing the Data – HTML, CSV, JSON Pandas – dataframes, concat, merge OpenrefineGoogleRefine –  Seemed to do trick so NLTK will come another day Natual Language Processing – Briefly…