We’ve always believed in giving back to the community. A lot of our projects involve R&D, and when we hit upon something that we think is a winner, we always share.
A flexible client-server crawling framework written in Go. It was written to act as a layer on top of a regular crawling library, with a lot of glue code pre-written. Apart from standard crawler features, it makes it easy to do things like scaling across machines, crawling through VPNs, using cookies, using Chrome as crawling backend, etc.