2024-08-30 Jerry Lundström Release 2.15.2 This releases fixes 3 issues detected by code analysis tools: - File not closed and memory not freed during error while loading known TLD file - https://github.com/DNS-OARC/dsc/security/code-scanning/22 label buffer should be static - https://github.com/DNS-OARC/dsc/security/code-scanning/20 unsigned difference expression 855f037 CodeQL b00cb11 Stack c4d3634 Sonar 26c3b9e Badges, fixes b4a9171 Workflow 2024-04-23 Jerry Lundström Release 2.15.1 This release fixes client subnet indexer which overwrote the mask options during initialization so the conf option `client_v4_mask` and `client_v6_mask` was never used. Other changes: - Update documentation - Update builtin known TLDs based on PSL - Update copyright year d577a97 Copyright f71edff Known TLDs dedafdd Client mask 8ef947c Doc 2023-08-09 Jerry Lundström Release 2.15.0 This release fixes DNS parsing w.r.t. EDNS, implements better loop detection during name decompression and adds a lot of EDNS indexers and filters. Previously the DNS parser expected the additional records to come straight after the question section. Meaning that if the DNS packet had any answer or authority records, they would be parsed as additional records for the OPT record and EDNS information. Following new indexers has been added: - edns_cookie - edns_cookie_len - edns_cookie_client - edns_cookie_server - edns_ecs - edns_ecs_family - edns_ecs_source_prefix - edns_ecs_scope_prefix - edns_ecs_address - edns_ecs_subnet - edns_ede - edns_ede_code - edns_ede_textlen - edns_ede_text - edns_nsid - edns_nsid_len - edns_nsid_data - edns_nsid_text Following new filters has been added: - edns0-only - edns0-cookie-only - edns0-nsid-only - edns0-ede-only - edns0-ecs-only See man-page dsc.conf(5) for more information. Other fixes/additions: - Only parse entire DNS message if EDNS indexers are used - `dns_protocol`: Implement proper loop detection during decompression - `xmalloc`: Check return of `amalloc()` before using `memset()`/`memcpy()` because it's undefined behavior on null pointers 8259f30 EDNS filters 41f3b9a strtohex, nsid text a666c04 EDNS(0) Client Subnet b5164fe EDNS 7cabfd9 EDNS0 parsing fixes and additional EDNS0 indexers. 46b1797 memcpy/memset fixes 8fd7b7a EDNS parsing cee2bf7 EDNS0 parsing, multi RR test a2c00c9 DNS compression loop detection 9875a3e RR parsing 2023-06-15 Jerry Lundström Release 2.14.1 Fixed a bug in TLD handling when using `tld_list`, it did not reset where it was in the QNAME when nothing was found and could therefor wrongly indicate something as a TLD. Also fixed a typo in the `dsc.conf` man-page. 976589d GCOV c3afee4 TLD list, doc typo 2023-04-03 Jerry Lundström Release 2.14.0 This release adds new configure option to control the file access to the output files, support for newer DNSTAP, improved DNSTAP message handling and updated Public Suffix List. - Fix #279: Add new conf options to control output file access: - `output_user`: set output file user ownership - `output_group`: set output file group ownership - `output_mod`: set output file mode bits - `dnstap`: Move DNSTAP essential attributes checks inside each type and customize them for that specific type - Update dnswire dependencies to v0.4.0 - `encryption_index`: Add support for new DNSTAP DNS-over-QUIC socket protocol - Update builtin Public Suffix List (PSL) abfe245 DNSTAP da06317 Output file access af01a48 DOQ transport, PSL update 2023-02-10 Jerry Lundström Release 2.13.2 Updated pcap-thread to v4.0.1: Fixed issue with `pcap_dispatch()` during non-threaded timed runs by checking packet timestamp and use `pcap_breakloop()` if the run should end. Based on reports, it looks like `pcap_dispatch()` won't stop processing if load is high enough even if documentation says "only one bufferful of packets is read at a time". Many thanks to Klaus Darilion @klaus3000 (NIC.AT) for the report and helping to track down the issue and test fixes! e7d92fe Fix COPR 7ecf217 pcap-thread 2022-04-21 Jerry Lundström Release 2.13.1 This patch release is mainly for build and packages where MaxMind DB library is preferred over the legacy GeoIP library. MaxMind has announced that the databases for GeoIP will be EOL May 2022 and recommends switching to GeoIP2 databases. Also updated DSC's description, removing references to the now discontinued Presenter and pointing to dsc-datatool instead. d891e2c Package, description c23406c Optional GeoIP 26dd506 GeoIP 2022-01-28 Jerry Lundström Release 2.13.0 This release fixes a huge performance issue with hashing IPv6 addresses, adds support for new DNSTAP messages types and protocols, and adds two new indexers. Thanks to a patch sent in by Ken Renard (@kdrenard) a rather huge performance issue related to hashing IPv6 addresses has been solved. Old code used a very incorrect assumption about addresses in general and while same way was used for IPv4, it didn't hit as hard as it did for IPv6. New code uses hashing functions on both address types and to quote the GitHub issue (by Ken): -"This performs about 5% better than what I did (51 sec versus 54 sec) for 5GB pcap file with nearly 50/50 split of IPv4 and IPv6 (3.7M/3.5M v4/v6 queries). Old inXaddr_hash() has been running for 75 minutes and is about 20% done. I say this is a winner!" Many thanks to Ken for pointing this out and supplying a patch! DSC now depends on dnswire v0.3.0 which includes new DNSTAP messages types and protocols that was recently added to DNSTAP's Protobuf definition. The new `UPDATE_QUERY` and `UPDATE_RESPONSE` messages types are now supported and are interpret as `AUTH_QUERY` and `AUTH_RESPONSE`. The new socket protocols for DOT, DOH and DNSCrypt are also supported and are interpret as TCP for indexers such as `ip_proto` and `transport`. To get stats on the encryption itself you can use the new indexer `encryption`. Two new indexers have been added: - `label_count`: Number of labels in the QNAME - `encryption`: Indicates whether the DNS message was carried over an encrypted connection or not, and if so over which. For example "unencrypted", "dot" (DNS-over-TLS), "doh" (DNS-over-HTTPS). Other changes: - `inX_addr`: Rework structure, separate IPv4 and IPv6 addresses - Fix some DNSTAP tests - `transport_index`: Fix typo in code documentation 37df703 DNSTAP update, encryption indexer d27171f Label count indexer 6932247 Adding labellen indexer which counts the number of labels in a DNS message 68cc9c7 New IP hashing 2022-01-13 Jerry Lundström Release 2.12.0 This release adds a new conf option `tld_list` to control what DSC considers are TLDs, and a script to convert the Public Suffix List to this format (see `man dsc-psl-convert` for more information). For example, using this option will allow DSC to gather statistics on domains like `co.uk` and `net.au` that would otherwise be counted as `uk` and `au`. The release also updates the man-pages, clarifying how to use multiple `interface` and other similar options. And removes the deprecated cron upload scripts. e779a87 Remove upload scripts 2880f93 PSL TLD list ea04022 Update Copyright and known TLDs 5cbc7a4 Output format b7e6c35 Doc e66dae4 dh_auto_test 6a3e817 debhelper 89d033f Bye Travis fa1c179 Mattermost 2020-10-20 Jerry Lundström Release 2.11.2 This release fixes a bug in `asn_indexer` that didn't enabled the usage of MaxMindDB after successful initiation. Other changes include a typo fix in `configure` and a lot of coverage tests. 395b11a Travis, configure ffea9ed Tests 8b0bebd Tests 09f8174 Config tests d1514d4 Coverage 66b018c Coverage, ASN indexer 2020-08-18 Jerry Lundström Release 2.11.1 This release fixes a 17-year old code cut&paste mistake in the classification indexer, until now it's been classifying funny query types based on the query class. This fix was sent in by Jim Hague (Sinodun), thanks Jim! Other changes are based on code analysis reports and setup for code coverage. 8d4763c Correct funny-qtype classification. a1dd55e getline 29bd143 Coverage 685e504 SonarCloud f759515 Badges 2020-06-01 Jerry Lundström Release 2.11.0 This release updates the built in known TLDs table and adds the optional configuration option `knowntlds_file` to, instead of using the built in table, load the data from a file. If compiled with only MaxMindDB support then ASN and Country indexer would complain (and exit) that no database has been specified. This release changes the behavior to match that of GeoIP support, making it possible to run without specifying a database. Other changes: - Fix compile warnings - COPR packaging fixes - `country_indexer`: Fixed typos in log messages (was copied from ASN) - Fix issues and false-positives reported by newer version of scan-build Commits: e937d1 COPR 1382370 country, asn 423a813 scanbuild 2571b97 Compile warnings 4f69447 Known TLDs 2020-05-07 Jerry Lundström Release 2.10.0 This release adds new configuration options to `dnstap_unixsock` to control ownership and permissions for the DNSTAP socket file. Other fixes: - Unlink the DNSTAP socket file if an error during initialization occur - Do hard exit in forks to not run `atexit()` (which will unlink the DNSTAP socket file) Commits: 9d1d49a fork 733b286 DNSTAP socket 2020-04-02 Jerry Lundström Release 2.9.1 This release fixes a few bugs, removes a lot of the debug messages about DNSTAP and removes GeoIP from openSUSE/SLE packages as it has been deprecated on those platforms. Changes: - `daemon`: Fix bug with listening for SIGINT when in foreground mode - `dnstap`: - Fix #217: Unlink UNIX socket on exit if successfully initiated - Fix startup bug, `exit()` if unable to initialize - Fix #220: - Remove/hide a lot of debug messages and the printing of the DNSTAP message - Clarify a lot of the info and error messages - Prefix all DNSTAP related messages with `DNSTAP: ` - Fix compile warnings and include headers when GeoIP is missing - `asn_indexer`: Fix bug, said unknown IPv4 when it was IPv6 Commits: 08bad5b DNSTAP debug 1232264 LGTM 589ea7a GeoIP, asn indexer 4fea0d2 sigint, DNSTAP UNIX socket, DNSTAP init 2020-03-20 Jerry Lundström Release 2.9.0 This release adds support for receiving DNS messages over DNSTAP along with documentation updates and eliminated compiler warnings. To enable DNSTAP support, install dependencies (check `README.md`) and run configure with `--enable-dnstap`. New configuration options: - `dnstap_file`: specify input from DNSTAP file - `dnstap_unixsock`: specify DNSTAP input from UNIX socket - `dnstap_tcp`: specify DNSTAP input from TCP connections (dsc listens) - `dnstap_udp`: specify DNSTAP input from UDP connections (dsc listens) - `dnstap_network`: specify network information in place of missing DNSTAP attributes Other changes: - Add documentation about extra configure options that might be needed for FreeBSD/OpenBSD - Fix compile warnings on FreeBSD 11.2 - Fix compile warning `snprintf()` truncation - Packaging updates Commits: 60e6950 DNSTAP af0417b README 1f1b489 COPR, spec 435e136 Package 3f24feb FreeBSD 11 compatibility 563b986 Funding 2019-04-23 Jerry Lundström Release 2.8.1 Added all missing config options for the response time indexer: - `response_time_mode` - `response_time_bucket_size` - `response_time_max_queries` - `response_time_full_mode` - `response_time_max_seconds` - `response_time_max_sec_mode` Commits: 36f0280 Response time config 2019-02-11 Jerry Lundström Release 2.8.0 This release brings an new indexer `response_time` (funded by NIC.AT!), support for MaxMind DB (GeoIP2) and an option to set the DNS port. The new indexer `response_time` can track queries and report the time it took to receive the response in buckets of microseconds or in logarithmic scales (see `response_time_mode`). It will also report timeouts, missing queries (received a response but have never seen the query), dropped queries (due to memory limitations) and internal errors. Here is an example output of log10 mode: New configuration options: - `asn_indexer_backend`: Control what backend to use for the ASN indexer - `country_indexer_backend`: Control what backend to use for the country indexer - `maxminddb_asn`: Specify database for ASN lookups using MaxMind DB - `maxminddb_country`: Specify database for country lookups using MaxMind DB - `dns_port`: Control the DNS port - `response_time_mode`: Set the output mode of the response time indexer - `response_time_bucket_size`: The size of bucket (microseconds) - Following options exists to control internal aspects of `response_time` indexer, see man-page for more information: - `response_time_max_queries` - `response_time_full_mode` - `response_time_max_seconds` - `response_time_max_sec_mode` Fixes: - Add LGTM and fix alerts - Update `pcap_layers` with fixes for `scan-build` warnings - Fix port in debug output of DNS message, was showing server port on responses Commits: f38a655 License 48cd44e Man-page, interface any, response time 8b9345f LGTM Alert e57a013 DNS port 38aa018 Response time statistics 7a60d53 Cleanup 5c45ce2 Copyright 0dc8a3c MaxMind DB (GeoIP2) 473387b LGTM, README, packages, scan-build 2018-08-14 Jerry Lundström Release 2.7.0 Add support for Linux "cooked" capture encapsulation (`DLT_LINUX_SLL`). Fixes: - `grok_question()`: Remove usage of `strcpy()` - `pcap_tcp_handler()`: Use `snprintf()` - `printable_dnsname()`: Use `snprintf()` - Fix CID 104450, 186871 Commits: 41d59ac man-page HTML 476d6ed pcap_layers, CID 747131b Configure options 43c9ad0 DLT_LINUX_SLL 8a48667 Support the linux cooked sll frame bd4a94f Fix CID 104450 2017-08-21 Jerry Lundström Release 2.6.1 Compatibility fixes for FreeBSD 11.1+ which is now packing `struct ip`. Commits: c0cd375 Handle compile warnings and FreeBSD's packing of structs c528ccb Code formatting and moved external code to own directory 2017-07-11 Jerry Lundström Release 2.6.0 Two new DNS filters and configuration for client subnet netmask has been added thanks to pull request submission from Manabu Sonoda (@mimuret), see `man 5 dsc.conf` for more details. New DNS filters: - `servfail-only`: Count only SERVFAIL responses - `authentic-data-only`: Count only DNS messages with the AD bit is set New configuration: - `client_v4_mask`: Set the IPv4 MASK for client_subnet INDEXERS - `client_v6_mask`: Set the IPv6 MASK for client_subnet INDEXERS Fixes: - Set `_DEFAULT_SOURCE`, was giving compile warnings on some platforms - Update `pcap-thread` to v2.1.3 for compatibility fixes - Fix bug where extra `"` would be OK in configuration - Eat all white-space between tokens in configuration - Minor documentation corrections Commits: 8a20421 Config parse quote/whitespace bug 4eb91d8 PR review and corrections 1dcdbc1 add supports statistics for DNSSEC validation resolver - SERVFAIL DNS message filter - AD bit DNS message filter - set custom mask for ClientSubnet 7c4ce7e Update pcap-thread to v2.1.3 f5d152c Corrected date 04f137d Prepare SPEC for OSB/COPR 402c242 Config header is generated by autotools 2017-03-29 Jerry Lundström Release 2.5.1 Various compatibility issues and a possible runtime bug, related to pcap-thread, fixed. Commits: 5ed03e3 Compat for OS X 8605759 Fix compiler warnings 5fbad26 Update pcap-thread to v2.1.2 47ed110 Update pcap-thread to v2.1.1 2017-03-02 Jerry Lundström Release 2.5.0 Resolved memory leaks within the IP fragment reassembly code that was reported by Klaus Darilion (NIC.AT) and added config option to control some parts of the fragment handling. Fixes: - Add `pcap_layers_clear_fragments()` to remove old fragments after `MAX_FRAG_IDLE` (60 seconds) - Use correct alloc/free functions for dataset hash - Fix spacing in dsc.conf(5) man-page New config option: - `drop_ip_fragments` will disable IP fragmentation reassembling and drop any IP packet that is a fragment (even the first) Commits: eaee6c0 Drop IP fragments 3ebb687 Issue #146: Fix leak in fragment handling 9a5e377 Use correct alloc/free 35f663c Fix #107: add const 2017-01-27 Jerry Lundström Release 2.4.0 Since there have been a few major issues with the threaded capturing code it is now default disabled and have to be enabled with a configure option to use: `./configure --enable-threads ...` A lot of work has been done to ensure stability and correct capturing, as of now `dsc` is continuously running on the testing platforms with simulated traffic and tests are performance every 5-15 minutes: https://dev.dns-oarc.net/jenkins/view/dsctest/ With the rewrite of the config parser to C it was missed that Hapy allowed CR/LF within the values of the options. Changing the C parser to allow it is a bit of work and having CR/LF within the value may lead to other issues so it is now documented that CR/LF are not allowed in config option values. Fixes: - The `-T` flag was just controlling pcap-thread usage of threads, it now controls all usage of threads including how signals are caught. - Fix program name, was incorrectly set so it would be reported as `/dsc`. - Use thread safe functions (_r). - Handle very long config lines by not having a static buffer, instead let `getline()` allocate as needed. - Use new activation in pcap-thread to activate the capturing of pcaps after the initial interval sync have been done during start-up. - Use factions of second for start-up interval sync and interval wait. - Fix memory leaks if config options was specified more then once. - Use new absolute timed run in pcap-thread to more exactly end capturing at the interval. - Fix config parsing, was checking for tab when should look for line feed. - Exit correctly during pcap-thread run to honor `dump_reports_on_exit`. - Use 100ms as default pcap-thread timeout, was 1s before but the old code used 250ms. - Various enhancements to logging of errors. New config options/features: - `pcap_buffer_size` can be used to increase the capture buffer within pcap-thread/libpcap, this can help mitigate dropped packets by the kernel during interval breaks. - `no_wait_interval` will skip the interval sync that happens during start-up and start capturing directly, the end of the interval will still be the modulus of the interval. - `pcap_thread_timeout` can be used to change the internal timeout use in pcap-thread to wait for packets (default 100ms). - Log non-fatal errors from pcap-thread w.r.t. setting the filter which can indicate that the filter is running in userland because lack of support or that it is too large for the kernel. Special thanks to: - Anand Buddhdev, RIPE NCC - Klaus Darilion, NIC.AT - Vincent Charrade, Nameshield Commits: ee59572 Fix #111, fix #116: Update pcap-thread to v2.0.0, remove debug code 64befef Update copyright year 40a1fb4 Fix #139: Use 100ms as default pcap-thread timeout 2a07185 Fix #137: Graceful exit on signal during run f1b3ec3 Issue #116: Try and make select issue more clear 950ea96 Fix #133: Return from `Pcap_run()` on signal/errors 667cc91 Issue #116: Add config option pcap_thread_timeout 3c9e073 Notice if non-fatal errors was detected during activation 4ea8f54 Fix #108: Document that CR/LF are not allowed within configuration line 9fda332 Check for LF and not tab 15a1dc0 Use pcap-thread timed run to interface 1e98f8b Fix potential memory leaks if config options specified more then once a9b38e9 Add missing LF and indicate what config option was wrong if possible f8a2821 Use fractions of seconds for both start up interval sync and timed run, always adjust for inter-run processing delay f47069a Fix #121: Update to pcap-thread latest develop fc13d73 Issue #116: Feature for not waiting on the interval sync c832337 Fix #122: Update pcap-thread to v1.2.3 for fix in timed run 4739111 Add `pcap_buffer_size` config option 7d9bf90 Update pcap-thread to v1.2.2 ef43335 Make threads optional and default disabled c2399cf getline() returns error on eof, don't report error if we are 5c671e6 Clarify config error message and report `getline()` error 8bd6a67 Fix #114: Handle very long lines 47b1e1a Use _r thread safe functions when possible 0f5d883 Update daemon.c f18e3ea Update doc, -T now disables all usage of threads 57aacbe Honor the -T flag when installing signal handlers 2016-12-22 Jerry Lundström Release 2.3.0 Rare lockup has been fixed that could happen if a signal was received in the wrong thread at the wrong time due to `pcap_thread_stop()` canceling and waiting on threads to join again. The handling of signals have been improved for threaded and non-threaded operations. A couple of bugfixes, one to fix loading of GeoIP ASN database and another to use the lowest 32 bits of an IP address (being v4 or v6) in the IP hash making it a bit more efficient for v6 addresses. New functionality for the configure option `local_address`, you can now specify a network mask (see `man 5 dsc.conf` for syntax). Commits: e286298 Fix CID 158968 Bad bit shift operation c15db43 Update to pcap-thread v1.2.1 1ac06ac Move stopping process to not require a packet 597dd34 Handle signals better with and without pthreads bcf99e8 Add RPM spec and ACLOCAL_AMFLAGS to build on CentOS 6 667fe69 fixed load geoIP ASN database from config-file e1304d4 Fix #97: Add optional mask to `local_address` so you can specify networks 5dae7dd Fix #96: Hash the lowest 32 bits of IP addresses 2016-10-15 Jerry Lundström Release 2.2.1 Two bugfixes with one being critical (caused segfault, sorry for that) - `pcap-thread` had an issue that threads where not closed on exit of `pcap_thread_run()`, this only created many threads in my testing environment but reports came in that it segfaulted. - When started `dsc` (not in debug mode) it will wait to align with the time, that did not get update with the configurable interval change. Commits: 6e3654b Fix #90: Update pcap-thread to v1.1.2 to fix a segfault issue c9350a3 Fix #92: Honor configured interval when aligning acaf617 minor typo 2016-10-10 Jerry Lundström Release 2.2.0 Some big changes in this release are the removal of the C++ configure parser library Hapy and the addition of pcap-thread to (hopefully) handle capturing packet in a correct and efficiant way. With that addition of pcap-thread comes new runtime options: - `-m` sets monitor mode on interfaces - `-i` sets immediate mode on interfaces - `-T` disable the usage of threads in pcap thread Bugfixes: - d95190a fixes a small memory leak in `Pcap_init()` and a possible bug where `-p` might never been used because of not being declared external. - 55e1056 added check for `netinet/ip_compat.h` for use in `src/pcap_layers/pcap_layers.c`. Commits: 0a1ce91 Fix coverity issues 5a1d410 Delete useless line (related to mayasd#84) 443db3e Check if the file was previously linked but not yet unlinked (Tim CLERC.IM) 02a7621 Fix #82: Oops, `pcap_thread_set_filter()` had changed during development and missed this 2a8aa29 Move definition of token struct inside and changed name to not conflict on FreeBSD 43da964 Fix #9: Implement conf parser in C and remove dependency of Hapy and C++ 9f46f0d Update pcap thread to version 1.1.1 d95190a Use pcap thread, new options `-miT` and possible bugfix for `-p` and a small memory leak fix 55e1056 Fix #77: Check for netinet/ip_compat 4e120f9 Fix travis script, only expand dir. 2016-06-28 Jerry Lundström Release 2.1.1 Bugfixes: - 22688c1 Fix pcap/select last_ts In some cases `select` will return the fd set as if there are packets to read but there aren't. That would case the last timestamp to not advance and the `Pcap_run` loop to never finish. This fix add a check on packets captured and sets last timestamp if none where caught. Commits: d115b3f Correct configuration, missing `;`. 22688c1 Fix pcap/select last_ts b6d3dd8 Fix package dependencies. c8979c4 Add debian/ubuntu package files 2016-06-09 Jerry Lundström Release 2.1.0 This release brings a couple of new features, the ability to change the interval for which DSC writes out the statistics files and a new indexer for AS numbers. There was also an issue detected in Ubuntu 16.04 when reading pcap files would result in doubling the statistics if running in daemon mode. Looking at the strace it might be a issue with the kernel, libc, pcap or a combination that results in the open file handle to reset and essentially read all the data twice. Credits: - Klaus Darilion NIC.AT for interval changes and AS number indexer. Commits: 8ab8632 Rewrote ASN indexer to remove the need for malloc/free. Removed the need to malloc ipstr. Use same unknown tags as country indexer. 89d4984 add ASN index: maps src-IP to AS number using GeoIP ASNum DB 8d5c6bf Fix #14: Remove ncap c0f00e7 Add test for statistics_interval cf9ede2 Add checks and documentation for statistics_interval 93eeecd new config option 'statistics_interval' (defaults to 60s) a28f5d4 Fix #62: Do not go into daemon mode if reading offline files 2bc1abb Add information about puppet module 2016-06-01 Jerry Lundström Release 2.0.0 This release brings a major update to the DSC software with the separation of the Collector and the Presenter, this repository will only include the Collector from now on. Please read UPGRADE.md for information on upgrading from previous version. Major changes / additions are: - Use of Automake and rework of the Makefiles - Conform to FHS 3.0 - Man-pages (man dsc, man dsc.conf) - Continuous Integration testing using Travis-CI - Compatibility testing on Debian, CentOS, FreeBSD and OpenBSD - Use of Coverity Scan to find defects - JSON output format, see output_format in dsc.conf(5). - IPv6 support in country indexer and libgeoip is now runtime configurable, see geoip_v4_dat / geoip_v6_dat in dsc.conf(5). - Signal handling and optional write reports on exit, see dump_reports_on_exit in dsc.conf(5). - Upload scripts are deprecated Credits: - Klaus Darilion NIC.AT for GeoIP IPv6 patch. - Michael Braunoeder NIC.AT for NXDOMAIN filter patch. - L-root for overflow bugfix IP fragments. - McStork for JSON output patch. Changes since release candidate: 1be5148 Fix #57: Flush the pid file to write it out and add test for pid file 0f79aa0 Use Semantic Versioning 2.0.0 semver.org 2016-05-10 Jerry Lundström Release 2.0.0-rc.1 69ef9b4 Add -v to display version 7e5b403 Fix defects 0f64128 Add badges f795ed3 Old automake needs AM_PROG_CC_C_O 27ae870 Fix #4: Remove old indexers, update indexers and filters documentation, update authors. d873411 Fix #38: Use locking to ensure we do not overwrite PID file 800fe83 Fix #35: GeoIP configurable. Fix debug and syslog in country indexer. Add documentation and config example. a1dcdf1 Avoid the checks for newer .dat files, this may have performance impacts. bb9a059 Add IPv6 support to country indexer and add some failure handling 03e16a8 Include config.h as this defines HAVE_LIBGEOIP. Thus, the geoip code was not used yet. 7808d2e New dsyslog/dsyslogf/dfprint macro. Change fprintf to dfprintf. fe47288 Fix #34: Create man-pages. 7a9b3c3 Fix #33: Handle most signals, new config option to dump reports on exit. 4753eda Add a filter to track NXDOMAIN responses 1565952 Add info about DSP 05ef699 Fix make test in dist. 68bdc9b No default interface in conf example 2be98be Correct libexec directory. Install etc files in subdirectory. 319ac4e Only build dist. Build in a build directory. 96e0e73 Fix #16: conform to FHS. Automake tweaks. bbcca74 Fix #23: Add missing changes for master branch 197ad52 No need to install autoconf/automake in Travis eb95ee1 Add license to cron scripts 4f62420 Fix/add make dist and try it in Travis ac4c634 Reconstruct repository to move out presenter. Update licenses. Use pcap_layers as a git submodule. Use automake/autoconf. 89c7f4c Import patched pcap_layers.c code with buffer overflow bugfix 2787db2 Only wait a certain number of times for the files to appear b5d911d Add the first test, simple run and compare gold files 32fd807 Fix #13: Document ability to read packets from pcap files eee217e Rework some of the Makefiles based on some of the patches found in the Debian packages 7a2a67e Fix #12: Add hash for dataset names and check for duplicates ed1eba9 Add base64 for certain non-printable characters in JSON output and remove extra new-line 580d543 Add output_format and JSON structure description to documentation d2d1ed2 Fix #3: Reworked JSON output format implementation 2e2f90f Give option for additional output JSON/Extended JSON a23b6af Ignore generated files f4214f3 Add Travis CI. Remove old TODAY/tar commands. Add ifndef/def to all .h files. Fix issue with arpa/nameser_compat.h on OpenBSD by checking for the header file and only including it if it exists. Rename configure.scan to configure.ac and change contact info and version. Update configure using autoconf 2.69 on Ubuntu 14.04 LTS. Set CC/CPP/CXX/CXXFLAGS in Makefile found by configure. 2016-01-11 Duane Wessels added dfprintf() macro to improve code readability and avoid multi-line if (debug_lvl) fprintf(...) statements. 2016-01-11 Duane Wessels Commenting out ancount and nscount to silence compiler warnings about unused variables. 2016-01-11 Duane Wessels Patch from John Heidemann relating to TCP reassembly - fix for multiple DNS messages per TCP connection that span multiple segments - some editorial code changes (== to >=) - additional debugging statements 2016-01-11 Duane Wessels Minor fix: debugging statement wasn't protected with debug_flag check. (thanks John H). 2015-12-23 Duane Wessels add prototype for dns_message_handle() 2015-12-23 Duane Wessels Experimental feature to drop "received responses" and "sent queries" since DSC is generally used to monitor authoritative servers that, under normal operation, never receive responses nor send queries. Currently protected with #ifdefs. 2015-12-15 Duane Wessels Oops, forgot to set dns_message->server_ip_addr for new server_addr indexer 2015-11-23 Duane Wessels removing one level of the "dsc/dsc" top directory 2015-11-11 aqadeer In pcap.c pcap_setnonblock doesn't accept captured files anymore and must need a device from where it could do live capture. For offline files to work, a simple check is added to by-pass this porblem. 2015-11-05 Duane Wessels Added a 'server' indexer. This records the server (query destination) IP address. 2015-11-05 Duane Wessels Renamed "client_ipv4" to "client_ip" because it supports v4 and v6 2015-11-04 Duane Wessels Update copy of pcap_layers.c from https://github.com/wessels/pcap_layers 2015-11-04 Duane Wessels avoid "void *" pointer arithmetic (Yoshitaka Aharen) 2015-10-09 Duane Wessels bugfix: handle receiving TCP dns length prefix out of order. A user reported that when a TCP segment containing only the DNS message length is received *after* the message it references (i.e., out-of-order), then DSC goes into a 100% CPU loop. Confirmed that the code doesn't work correctly when dnslen comes out-of-order, but I wasn't able to easily reproduce the 100% CPU bug. The fix is to add a "dnslen_bytes_seen_mask" variable that tracks which of the two dnslen bytes we've seen. Once both bytes have been seen, then we can proceed to reassembling the message buffer. 2015-10-09 Duane Wessels cosmetic: rename ipv4 to ip4 to be consistent with ip6 2015-10-09 Duane Wessels Regarding recent change to add pcap_layers library, forgot to add it to Makefile.in (vs Makefile) before committing. 2015-10-09 Duane Wessels Integrated https://github.com/wessels/pcap_layers with DSC collector. The DSC code now includes a copy of the pcap_layers library, which does a better job of extracting the layers of a pcap packet. In particular, it does IP fragmentation reassembly, which is important for DSC and RSSAC-002. At this time the DSC code still does TCP reassembly, however. Removed USE_IPV6 ifdefs. IPV6 support is now always compiled. 2015-10-08 Duane Wessels Removing "DMC *dns_message_callback" because there is only one that would ever be used. The callback layer of indirection makes the code a little confusing, and also this change is in preparation for bringing in a third-party pcap layer handling library which will work slightly differently. 2015-10-08 Duane Wessels The "ip_message" code has not been in use since a commit back in 2012. But the code lingered, until now. 2015-02-25 Duane Wessels User reported a concern with the way dsc-xml-extractor.pl called the operating system 'mv' command. It has been replaced with Perl's File::Copy::mv(). 2013-03-19 Duane Wessels Fixed a bug in TCP reassembly when the DNS length field was split between two segments. 2012-08-27 Sebastian Castro Added 'tc_bit' indexer and dataset to track the frequency of responses having the TC bit set. Useful with DNSSEC signed zones. 2012-02-29 Duane Wessels Added 'qr_aa_bits' dataset and graph. It shows the distribution of QR/AA values in received messages and may be helpful in detecting reflector attacks targeting your name server. 2010-11-29 Duane Wessels The dnssec_qtype and dns_ip_version plots were not working due the presence of 'dataset' attributes in their definitions. Also fixed the chaos plot. 2010-11-04 Duane Wessels Added NSEC3 record type to the extractor. 2010-09-14 Duane Wessels A fix to put pcap interfaces into non-blocking mode, which is important when reading from multiple interfaces. 2010-08-13 Alexander Mayrhofer Added "country_index" to the collector which is an index based on country as returned by the GeoIP library. configure attempts to learn if you have GeoIP installed. 2010-08-13 Peter Koch Some minor fixes to the collector that allow it to read from pcap files on disk, rather than live packets from an interface. 2010-06-02 Henrik Kramsh?j Improvements for OpenBSD compatibility 2010-05-01 Duane Wessels Added NSEC3 to DNSSEC query types 2010-02-22 Duane Wessels Collector bug fixed: USE_IPV6=1 was not passed to .c files, which resulted in different-sized data structurs and crashes on CentOS. 2009-12-10 Duane Wessels Added "priming_queries" and "priming_responses" datasets in preparation for root zone signing. 2009-11-11 Duane Wessels On the presenter some of the "accum" graphs stopped working due to recent rewrites. These should now be working again. 2009-10-12 Duane Wessels The presenter debugging is now configurable via dsc-grapher.cfg. 2009-08-14 Duane Wessels In the presenter, there have been some significant changes to the perl modules so that they can be used in command line (ie non-CGI) utilities, both for creating graphs and for reading data. Most of these changes are purely internal. However, you may need to update your dsc-grapher.pl CGI program to be like presenter/grapher/dsc-grapher.pl.sample in the source distribution. 2009-04-15 Duane Wessels In the presenter's refile-and-grok.sh script, note that in some environtments it is useful to skip NODEs that don't have an incoming directory because they might have been "grokked" elsewhere and then rsync'd to you. 2009-02-27 Duane Wessels In the collector it is now necessary to include and . Not sure why it wasn't necessary before. Perhaps due to 2009-01-26 Hapy library upgrade. 2009-02-19 Duane Wessels In the collector, fixed some 64-bit free disk space calculations with casting. 2009-01-26 Duane Wessels Upgraded to more recent version of Hapy parsing library. 2008-12-31 Duane Wessels In the collector, changed the value of pcap_open_live() because we always try to read from all interfaces after select() returns. If we have multiple interfaces and one of them is quiet, and the timeout is too large, we'll drop packets on the other interface while waiting for the first to time out. Also added a pseudo-dataset that reports on pcap packets captured and dropped. 2008-12-02 Duane Wessels Grapher now preserves the order of the server list given in dsc-grapher.cfg. 2008-11-22 Duane Wessels Added some collector indexers and datasets for 2nd and 3rd level domains. 2008-10-02 Duane Wessels Made the 'no extractor for $dataset' error message non-fatal. Now, the unknown dataset will just be skipped and the remaining ones will be processed. 2008-09-30 Duane Wessels In the presenter, account for the possibility that the collector's clock is not in sync. 2008-09-18 Elmar Knipp Found bug in upload script where a $PROG.out prevents removal of a $YYYYMMDD directory. 2008-09-16 Duane Wessels New presenter feature: Node Merging. Now the DSC grapher can automatically "merge" multiple nodes so that they appear as one. For example instead of this dsc-grapher.cfg line: server S N1a N1b you can merge nodes "a" and "b" with this line: server S N1=N1a,N1b This feature is also useful for just renaming a node if you want it to be displayed as a different name than the directory where its files are. 2008-09-15 Duane Wessels On the collector, changed the sample config and upload scripts to use /usr/local/dsc/run as the default run_dir (was formerly /usr/local/dsc/var/run/). 2008-08-21 Duane Wessels Added an EDNS bufsiz indexer so we can collect buffer sizes advertised by clients. 2008-08-12 Duane Wessels Attempt to improve portability (on Solaris) by checking for libresolv, libnsl, and libsocket. Also check for statvfs() vs statvs() in configure. 2008-07-29 Duane Wessels Updates to collector/cron/upload-rsync.sh so that it behaves better with new date-based directories on both collector and presenter. 2008-06-27 Duane Wessels Added a source port range dataset (for about-to-be-announced vulnerability that can be mititgated by improving source port randomness). 2008-04-22 Duane Wessels Added ./configure script for presenter/dsc code. 2008-04-22 Duane Wessels Added support for capturing NCAP data streams. 2008-01-09 Duane Wessels Changed the location of the presenter's ".dat" files. Previously there were stored in a directory such as 20080109/qtype/qtype.dat and now they will be moved one directory up and stored as 20080109/qtype.dat. The source distribution includes a script found at presenter/grapher/update-dat-file-locations.sh that will traverse the /usr/local/dsc/data directory and move all the .dat files one level up. When upgrading to this version of DSC you should first stop the refile-and-grok.sh cron job, install the DSC software, and then run the update-dat-file-locations.sh script (after reading it). 2008-01-09 Duane Wessels Data passed from collector to presenter is now contained in a single XML file, rather than one XML file per dataset. This should significantly reduce filesystem pressure (i.e., 20x fewer files to be stored and sent) on both sides. 2008-01-07 Duane Wessels The collector now stores to-be-uploaded XML files in date-named subdirectories. This reduces filesystem pressure in the event that the collector cannot communicate with the presenter for an extended period of time. To accomodate this change, the upload-prep.sh script has been replaced with a Perl version, named upload-prep.pl. After installing DSC you should change your cron job to use the new script. 2008-01-04 Ken Keys Previously, a child was forked for each collection interval. Memory allocated for collected data was freed by simply exiting the child. But this left no way to preserve any data across intervals. Now, interval collectors do not fork. To avoid the tedious, slow, and error-prone nature of normal malloc-and-free memory management, we instead store non-persistent data in memory allocated from an "arena", which can be freed quickly and safely. We don't need to free individual allocations, but we do need to reset some pointers to allocated memory. TCP resets are now properly handled and TCP state is freed if a connection has been idle for too long. 2008-01-04 Duane Wessels The collector configuration (dsc.conf) has a new 'minfree_bytes' directive. If the amount of free disk space on dsc's partition falls below this limit, dsc will not write any XML files -- that data will be lost. The default value is 5 MB. 2007-12-14 Alexander Gall Added a 'dns_ip_version' indexer so that we can track DNS messages sent over v4 and v6. 2007-12-04 Duane Wessels Fixed a collector bug when listening on multiple interfaces of different datalink types (e.g., eth0 and lo). 2007-11-26 Duane Wessels Added a 'hide_nodes' feature to dsc-grapher.cfg. If 'hide_nodes' is given, then the navigation menu will not display the nodes underneath a server. Note, however, that knowledgeable users could still view individual node data by manually if they know the names by specifying it in the URL query terms. 2007-10-09 Duane Wessels Changed the presenter to not use "data URIs" by default. Now Internet Explorer users should be happier. 2007/06/14 Duane Wessels Added -f command line option to keep 'dsc' collector as a foreground process. 2007-06-11 Duane Wessels dsc-xml-extractor.pl now looks for incoming XML files in "incoming/YYYY-MM-DD" directories. 2007-06-07 Duane Wessels New TCP code did not compile cleanly on Linux. Added some #ifdefs and macros for Linux compatibility on TCP headers. 2007-05-15 Ken Keys, Duane Wessels TCP support New 'transport' indexer for DNS messages. Indicates whether message arrived over UDP or TCP. 2007-04-04 Duane Wessels IPv6 support New 'ip_version' indexer. Indicates whether DNS message arrived via IPv4 or IPv6.