2024-12-02 | Started FCC data ingestor | 1 | +216 | -0 |
2024-11-30 | Added documentation to Meshtastic ingestor on how to collect MQTT evens in JSON | 1 | +12 | -1 |
2024-11-30 | Added mapping for Meshtastic MQTT events, improved payload processing | 1 | +127 | -6 |
2024-11-30 | removed typings import | 3 | +157 | -8 |
2024-11-30 | Added meshtastic & firehol ingestors | 2 | +296 | -0 |
2024-11-27 | Improved processing using the hidden -oD output for ndjson | 1 | +21 | -34 |
2024-11-24 | RIR transfers ingestion fixed & tested | 1 | +35 | -3 |
2024-11-22 | Refactoring RIR ingestions | 4 | +71 | -35 |
2024-03-24 | ECS formatting added to eris, certstream ingestor now caches to prevent duplication, elastic connections properly closed now | 5 | +60 | -21 |
2024-03-23 | Fixed _doc to doc in record | 1 | +7 | -7 |
2024-03-23 | Added more comments and thoughts | 1 | +2 | -1 |
2024-03-23 | Added a comment about the methodology of determining the zone the data is from. | 1 | +1 | -0 |
2024-03-23 | Updated example record comment under source to reflect on the new record structure | 1 | +4 | -1 |
2024-03-23 | AXFR detection in zone file input, to determine if the data is form ICANN or an AXFR against a dns (for source tracking) | 1 | +10 | -2 |
2024-03-23 | Added zone field too better querying on specific tlds/zones | 1 | +5 | -0 |
2024-03-23 | Records stored as a nested type for better querying, added source field so we can identify where the zone data derived (icann, axfr, breach, etc) | 1 | +9 | -6 |
2024-03-19 | Added IXP ingestor, updated all other ingestors | 7 | +476 | -326 |
2024-03-15 | Typo in pypi title | 1 | +1 | -1 |
2024-03-15 | Added elastic common schema (ecs) logging to file for ingesting eris logs straight into ES | 2 | +61 | -54 |
2024-03-14 | Certstream ingestor now only logs sub-domains since we already ingested zone files. Ignores www. and wildcard domains. | 1 | +14 | -4 |
2024-03-14 | add ingest_rir_transfers.py | 1 | +7 | -13 |
2024-03-14 | RIR Transfers ingestor added | 1 | +193 | -0 |
2024-03-13 | Thats how you fucking do it. Delegations stored in ranges now for querying. | 1 | +34 | -22 |
2024-03-12 | Fixed import error saying aiofiles instead of aiohttp | 1 | +2 | -2 |
2024-03-12 | Added anomaly detection to RIR delegations ingestor | 2 | +58 | -41 |
2024-03-12 | Created an ingestor for RIR delegations | 1 | +167 | -0 |
2024-03-12 | Lowered reconnect time on certstream timeout | 1 | +1 | -1 |
2024-03-12 | Renamed default_index to eris-certstream | 1 | +1 | -1 |
2024-03-12 | Fixed logger again | 2 | +12 | -9 |
2024-03-12 | Fixed missing logging.handlers import | 1 | +1 | -0 |
2024-03-12 | Fixed logger level | 1 | +1 | -1 |
2024-03-12 | Removed _id from certstream yield and renamed ingest_certs.py to ingest_certstream.py | 6 | +492 | -465 |
2024-03-12 | Code cleanup | 3 | +397 | -468 |
2024-03-11 | Full source commenting, uniformity in testing function, records stored as a list by default incase an IP address yields multiple PTR records | 1 | +75 | -71 |
2024-03-08 | Many bugs fixed in sniffer and async model. | 3 | +52 | -49 |
2024-03-08 | Overall code cleanup | 4 | +50 | -45 |
2024-03-08 | Fixed issue with ingest_certs and the ingestion function signature. Simple placeholder argument (un-used) added to maintain function uniformity | 1 | +1 | -3 |
2024-03-08 | Testing function added to every ingestor to debug directly. No more --dry-run needed. | 6 | +274 | -194 |
2024-03-08 | Changed yield to return when sentinal value is recieved through the FIFO | 1 | +1 | -1 |
2024-03-08 | Added a testing function for calling the ingest_massdns.py file directly for debugging | 1 | +47 | -26 |
2024-03-08 | MassDNS ingestion script now caches the previous record to support IP addresses that yield more than one PTR record (field turned into a list when +1). Records will now upsert so MassDNS can be streaming into ES 24/7= | 1 | +0 | -4 |
2024-03-08 | MassDNS ingestion script now caches the previous record to support IP addresses that yield more than one PTR record (field turned into a list when +1). Records will now upsert so MassDNS can be streaming into ES 24/7= | 12 | +133 | -917 |
2024-03-06 | OCD about formatting again | 4 | +83 | -60 |
2024-03-06 | Certstream ingestion improved, still need to test stripping nulls and refactor the index mapping so we only store whats needed | 1 | +6 | -4 |
2024-03-06 | Updated massdns ingestion script with sentinal value checking and using the ip address as the document id | 1 | +32 | -25 |
2024-03-06 | Added a note about NSEC3 hash cracking for zone file data | 1 | +5 | -0 |
2024-03-06 | Zone file ingestion script now uses the same sentinal value as masscans ingestion, set document id as the domain name to allow updating records if they exist | 1 | +27 | -15 |
2024-03-06 | Masscan ingestion script updated to use ip:port as the document id to allow updating records that already exist. Added a sentinal value to trigger an EOF when using --watch with FIFO's | 1 | +60 | -57 |
2024-03-06 | Updated cause I am OCD about spaces and formatting | 1 | +6 | -6 |
2024-03-06 | Introduction paragraph descriving the project overview added, updated roadmap | 1 | +5 | -3 |
2024-03-06 | Asyncronous refactorization pushed as main version 💯 | 20 | +155 | -997 |
2024-03-06 | Added a recursive JSON explorer to remove empty or nulled keys from a dictionary | 2 | +31 | -2 |
2024-03-06 | Asyncronous refactorization of the codebase is complete, testing & metrics and then it will be production ready | 6 | +381 | -43 |
2024-03-05 | Asyncronous developed mon ERIS is complete, need to refactor ingestion helpers before pushing this as the main version | 5 | +209 | -222 |
2024-03-05 | Added certificate transparency monitoring via websockets for ingestion | 4 | +3 | -1 |
2024-03-05 | Added certificate transparency monitoring via websockets for ingestion | 4 | +191 | -3 |
2024-03-04 | Started asyncronous implementation of bulk streaming data, altered ERIS defaults, etc | 11 | +626 | -200 |
2024-02-02 | Added an asyncronous sniffer patch in helpers | 1 | +96 | -0 |
2024-02-02 | Ingestion agents are now modular, FIFO live ingestion added | 12 | +1059 | -1506 |
2024-01-27 | Updated README, copied over consistencies across the ingestors, docstring updates to reflect on new arguments | 6 | +455 | -187 |
2024-01-27 | Added parallel bulk uploading, error handling, sniffing nodes for discovery, dynamic batch sizes, and more | 1 | +0 | -0 |
2024-01-27 | Added parallel bulk uploading, error handling, sniffing nodes for discovery, dynamic batch sizes, and more | 7 | +527 | -163 |
2024-01-21 | Updated mirrors | 2 | +3 | -4 |
2024-01-20 | Updated README, fixed issue using the wrong domain in records for zone file ingestion (woops) | 6 | +75 | -21 |
2024-01-20 | Initial commit | 6 | +968 | -0 |