diff options
Diffstat (limited to 'CHANGES')
| -rw-r--r-- | CHANGES | 1288 |
1 files changed, 1288 insertions, 0 deletions
@@ -0,0 +1,1288 @@ +2024-08-30 Jerry Lundström + + Release 2.15.2 + + This releases fixes 3 issues detected by code analysis tools: + - File not closed and memory not freed during error while loading known + TLD file + - https://github.com/DNS-OARC/dsc/security/code-scanning/22 + label buffer should be static + - https://github.com/DNS-OARC/dsc/security/code-scanning/20 + unsigned difference expression + + 855f037 CodeQL + b00cb11 Stack + c4d3634 Sonar + 26c3b9e Badges, fixes + b4a9171 Workflow + +2024-04-23 Jerry Lundström + + Release 2.15.1 + + This release fixes client subnet indexer which overwrote the mask + options during initialization so the conf option `client_v4_mask` and + `client_v6_mask` was never used. + + Other changes: + - Update documentation + - Update builtin known TLDs based on PSL + - Update copyright year + + d577a97 Copyright + f71edff Known TLDs + dedafdd Client mask + 8ef947c Doc + +2023-08-09 Jerry Lundström + + Release 2.15.0 + + This release fixes DNS parsing w.r.t. EDNS, implements better loop + detection during name decompression and adds a lot of EDNS indexers + and filters. + + Previously the DNS parser expected the additional records to come + straight after the question section. Meaning that if the DNS packet + had any answer or authority records, they would be parsed as additional + records for the OPT record and EDNS information. + + Following new indexers has been added: + - edns_cookie + - edns_cookie_len + - edns_cookie_client + - edns_cookie_server + - edns_ecs + - edns_ecs_family + - edns_ecs_source_prefix + - edns_ecs_scope_prefix + - edns_ecs_address + - edns_ecs_subnet + - edns_ede + - edns_ede_code + - edns_ede_textlen + - edns_ede_text + - edns_nsid + - edns_nsid_len + - edns_nsid_data + - edns_nsid_text + + Following new filters has been added: + - edns0-only + - edns0-cookie-only + - edns0-nsid-only + - edns0-ede-only + - edns0-ecs-only + + See man-page dsc.conf(5) for more information. + + Other fixes/additions: + - Only parse entire DNS message if EDNS indexers are used + - `dns_protocol`: Implement proper loop detection during decompression + - `xmalloc`: Check return of `amalloc()` before using `memset()`/`memcpy()` because it's undefined behavior on null pointers + + 8259f30 EDNS filters + 41f3b9a strtohex, nsid text + a666c04 EDNS(0) Client Subnet + b5164fe EDNS + 7cabfd9 EDNS0 parsing fixes and additional EDNS0 indexers. + 46b1797 memcpy/memset fixes + 8fd7b7a EDNS parsing + cee2bf7 EDNS0 parsing, multi RR test + a2c00c9 DNS compression loop detection + 9875a3e RR parsing + +2023-06-15 Jerry Lundström + + Release 2.14.1 + + Fixed a bug in TLD handling when using `tld_list`, it did not reset + where it was in the QNAME when nothing was found and could therefor + wrongly indicate something as a TLD. + + Also fixed a typo in the `dsc.conf` man-page. + + 976589d GCOV + c3afee4 TLD list, doc typo + +2023-04-03 Jerry Lundström + + Release 2.14.0 + + This release adds new configure option to control the file access to + the output files, support for newer DNSTAP, improved DNSTAP message + handling and updated Public Suffix List. + + - Fix #279: Add new conf options to control output file access: + - `output_user`: set output file user ownership + - `output_group`: set output file group ownership + - `output_mod`: set output file mode bits + - `dnstap`: Move DNSTAP essential attributes checks inside each type and customize them for that specific type + - Update dnswire dependencies to v0.4.0 + - `encryption_index`: Add support for new DNSTAP DNS-over-QUIC socket protocol + - Update builtin Public Suffix List (PSL) + + abfe245 DNSTAP + da06317 Output file access + af01a48 DOQ transport, PSL update + +2023-02-10 Jerry Lundström + + Release 2.13.2 + + Updated pcap-thread to v4.0.1: + + Fixed issue with `pcap_dispatch()` during non-threaded timed runs by + checking packet timestamp and use `pcap_breakloop()` if the run + should end. + Based on reports, it looks like `pcap_dispatch()` won't stop + processing if load is high enough even if documentation says "only + one bufferful of packets is read at a time". + + Many thanks to Klaus Darilion @klaus3000 (NIC.AT) for the report + and helping to track down the issue and test fixes! + + e7d92fe Fix COPR + 7ecf217 pcap-thread + +2022-04-21 Jerry Lundström + + Release 2.13.1 + + This patch release is mainly for build and packages where MaxMind DB + library is preferred over the legacy GeoIP library. + MaxMind has announced that the databases for GeoIP will be EOL May 2022 + and recommends switching to GeoIP2 databases. + + Also updated DSC's description, removing references to the now + discontinued Presenter and pointing to dsc-datatool instead. + + d891e2c Package, description + c23406c Optional GeoIP + 26dd506 GeoIP + +2022-01-28 Jerry Lundström + + Release 2.13.0 + + This release fixes a huge performance issue with hashing IPv6 + addresses, adds support for new DNSTAP messages types and protocols, + and adds two new indexers. + + Thanks to a patch sent in by Ken Renard (@kdrenard) a rather huge + performance issue related to hashing IPv6 addresses has been solved. + Old code used a very incorrect assumption about addresses in general + and while same way was used for IPv4, it didn't hit as hard as it did + for IPv6. + New code uses hashing functions on both address types and to quote + the GitHub issue (by Ken): + -"This performs about 5% better than what I did (51 sec versus 54 sec) + for 5GB pcap file with nearly 50/50 split of IPv4 and IPv6 (3.7M/3.5M + v4/v6 queries). + Old inXaddr_hash() has been running for 75 minutes and is about 20% + done. I say this is a winner!" + + Many thanks to Ken for pointing this out and supplying a patch! + + DSC now depends on dnswire v0.3.0 which includes new DNSTAP messages + types and protocols that was recently added to DNSTAP's Protobuf + definition. + The new `UPDATE_QUERY` and `UPDATE_RESPONSE` messages types are + now supported and are interpret as `AUTH_QUERY` and `AUTH_RESPONSE`. + The new socket protocols for DOT, DOH and DNSCrypt are also supported + and are interpret as TCP for indexers such as `ip_proto` and + `transport`. To get stats on the encryption itself you can use the + new indexer `encryption`. + + Two new indexers have been added: + - `label_count`: Number of labels in the QNAME + - `encryption`: Indicates whether the DNS message was carried over an + encrypted connection or not, and if so over which. For example + "unencrypted", "dot" (DNS-over-TLS), "doh" (DNS-over-HTTPS). + + Other changes: + - `inX_addr`: Rework structure, separate IPv4 and IPv6 addresses + - Fix some DNSTAP tests + - `transport_index`: Fix typo in code documentation + + 37df703 DNSTAP update, encryption indexer + d27171f Label count indexer + 6932247 Adding labellen indexer which counts the number of labels in a DNS message + 68cc9c7 New IP hashing + +2022-01-13 Jerry Lundström + + Release 2.12.0 + + This release adds a new conf option `tld_list` to control what DSC + considers are TLDs, and a script to convert the Public Suffix List to + this format (see `man dsc-psl-convert` for more information). + + For example, using this option will allow DSC to gather statistics on + domains like `co.uk` and `net.au` that would otherwise be counted as + `uk` and `au`. + + The release also updates the man-pages, clarifying how to use multiple + `interface` and other similar options. And removes the deprecated cron + upload scripts. + + e779a87 Remove upload scripts + 2880f93 PSL TLD list + ea04022 Update Copyright and known TLDs + 5cbc7a4 Output format + b7e6c35 Doc + e66dae4 dh_auto_test + 6a3e817 debhelper + 89d033f Bye Travis + fa1c179 Mattermost + +2020-10-20 Jerry Lundström + + Release 2.11.2 + + This release fixes a bug in `asn_indexer` that didn't enabled the usage + of MaxMindDB after successful initiation. Other changes include a typo + fix in `configure` and a lot of coverage tests. + + 395b11a Travis, configure + ffea9ed Tests + 8b0bebd Tests + 09f8174 Config tests + d1514d4 Coverage + 66b018c Coverage, ASN indexer + +2020-08-18 Jerry Lundström + + Release 2.11.1 + + This release fixes a 17-year old code cut&paste mistake in the + classification indexer, until now it's been classifying funny query + types based on the query class. This fix was sent in by Jim Hague + (Sinodun), thanks Jim! + + Other changes are based on code analysis reports and setup for code + coverage. + + 8d4763c Correct funny-qtype classification. + a1dd55e getline + 29bd143 Coverage + 685e504 SonarCloud + f759515 Badges + +2020-06-01 Jerry Lundström + + Release 2.11.0 + + This release updates the built in known TLDs table and adds the optional + configuration option `knowntlds_file` to, instead of using the built in + table, load the data from a file. + + If compiled with only MaxMindDB support then ASN and Country indexer + would complain (and exit) that no database has been specified. + This release changes the behavior to match that of GeoIP support, + making it possible to run without specifying a database. + + Other changes: + - Fix compile warnings + - COPR packaging fixes + - `country_indexer`: Fixed typos in log messages (was copied from ASN) + - Fix issues and false-positives reported by newer version of scan-build + + Commits: + e937d1 COPR + 1382370 country, asn + 423a813 scanbuild + 2571b97 Compile warnings + 4f69447 Known TLDs + +2020-05-07 Jerry Lundström + + Release 2.10.0 + + This release adds new configuration options to `dnstap_unixsock` to + control ownership and permissions for the DNSTAP socket file. + + Other fixes: + - Unlink the DNSTAP socket file if an error during initialization occur + - Do hard exit in forks to not run `atexit()` (which will unlink the + DNSTAP socket file) + + Commits: + 9d1d49a fork + 733b286 DNSTAP socket + +2020-04-02 Jerry Lundström + + Release 2.9.1 + + This release fixes a few bugs, removes a lot of the debug messages + about DNSTAP and removes GeoIP from openSUSE/SLE packages as it has + been deprecated on those platforms. + + Changes: + - `daemon`: Fix bug with listening for SIGINT when in foreground mode + - `dnstap`: + - Fix #217: Unlink UNIX socket on exit if successfully initiated + - Fix startup bug, `exit()` if unable to initialize + - Fix #220: + - Remove/hide a lot of debug messages and the printing of the DNSTAP message + - Clarify a lot of the info and error messages + - Prefix all DNSTAP related messages with `DNSTAP: ` + - Fix compile warnings and include headers when GeoIP is missing + - `asn_indexer`: Fix bug, said unknown IPv4 when it was IPv6 + + Commits: + 08bad5b DNSTAP debug + 1232264 LGTM + 589ea7a GeoIP, asn indexer + 4fea0d2 sigint, DNSTAP UNIX socket, DNSTAP init + +2020-03-20 Jerry Lundström + + Release 2.9.0 + + This release adds support for receiving DNS messages over DNSTAP along + with documentation updates and eliminated compiler warnings. + + To enable DNSTAP support, install dependencies (check `README.md`) and + run configure with `--enable-dnstap`. + + New configuration options: + - `dnstap_file`: specify input from DNSTAP file + - `dnstap_unixsock`: specify DNSTAP input from UNIX socket + - `dnstap_tcp`: specify DNSTAP input from TCP connections (dsc listens) + - `dnstap_udp`: specify DNSTAP input from UDP connections (dsc listens) + - `dnstap_network`: specify network information in place of missing DNSTAP attributes + + Other changes: + - Add documentation about extra configure options that might be needed for FreeBSD/OpenBSD + - Fix compile warnings on FreeBSD 11.2 + - Fix compile warning `snprintf()` truncation + - Packaging updates + + Commits: + 60e6950 DNSTAP + af0417b README + 1f1b489 COPR, spec + 435e136 Package + 3f24feb FreeBSD 11 compatibility + 563b986 Funding + +2019-04-23 Jerry Lundström + + Release 2.8.1 + + Added all missing config options for the response time indexer: + - `response_time_mode` + - `response_time_bucket_size` + - `response_time_max_queries` + - `response_time_full_mode` + - `response_time_max_seconds` + - `response_time_max_sec_mode` + + Commits: + 36f0280 Response time config + +2019-02-11 Jerry Lundström + + Release 2.8.0 + + This release brings an new indexer `response_time` (funded by NIC.AT!), + support for MaxMind DB (GeoIP2) and an option to set the DNS port. + + The new indexer `response_time` can track queries and report the time + it took to receive the response in buckets of microseconds or in + logarithmic scales (see `response_time_mode`). It will also report + timeouts, missing queries (received a response but have never seen the + query), dropped queries (due to memory limitations) and internal errors. + + Here is an example output of log10 mode: + + <array name="response_time" dimensions="2" start_time="1478727151" + stop_time="1478727180"> + <dimension number="1" type="All"/> + <dimension number="2" type="ResponseTime"/> + <data> + <All val="ALL"> + <ResponseTime val="100000-1000000" count="77"/> + <ResponseTime val="10000-100000" count="42"/> + <ResponseTime val="1000-10000" count="3"/> + <ResponseTime val="missing_queries" count="1"/> + </All> + </data> + </array> + + New configuration options: + - `asn_indexer_backend`: Control what backend to use for the ASN indexer + - `country_indexer_backend`: Control what backend to use for the + country indexer + - `maxminddb_asn`: Specify database for ASN lookups using MaxMind DB + - `maxminddb_country`: Specify database for country lookups using + MaxMind DB + - `dns_port`: Control the DNS port + - `response_time_mode`: Set the output mode of the response time indexer + - `response_time_bucket_size`: The size of bucket (microseconds) + - Following options exists to control internal aspects of `response_time` + indexer, see man-page for more information: + - `response_time_max_queries` + - `response_time_full_mode` + - `response_time_max_seconds` + - `response_time_max_sec_mode` + + Fixes: + - Add LGTM and fix alerts + - Update `pcap_layers` with fixes for `scan-build` warnings + - Fix port in debug output of DNS message, was showing server port + on responses + + Commits: + f38a655 License + 48cd44e Man-page, interface any, response time + 8b9345f LGTM Alert + e57a013 DNS port + 38aa018 Response time statistics + 7a60d53 Cleanup + 5c45ce2 Copyright + 0dc8a3c MaxMind DB (GeoIP2) + 473387b LGTM, README, packages, scan-build + +2018-08-14 Jerry Lundström + + Release 2.7.0 + + Add support for Linux "cooked" capture encapsulation (`DLT_LINUX_SLL`). + + Fixes: + - `grok_question()`: Remove usage of `strcpy()` + - `pcap_tcp_handler()`: Use `snprintf()` + - `printable_dnsname()`: Use `snprintf()` + - Fix CID 104450, 186871 + + Commits: + 41d59ac man-page HTML + 476d6ed pcap_layers, CID + 747131b Configure options + 43c9ad0 DLT_LINUX_SLL + 8a48667 Support the linux cooked sll frame + bd4a94f Fix CID 104450 + +2017-08-21 Jerry Lundström + + Release 2.6.1 + + Compatibility fixes for FreeBSD 11.1+ which is now packing `struct ip`. + + Commits: + c0cd375 Handle compile warnings and FreeBSD's packing of structs + c528ccb Code formatting and moved external code to own directory + +2017-07-11 Jerry Lundström + + Release 2.6.0 + + Two new DNS filters and configuration for client subnet netmask has been + added thanks to pull request submission from Manabu Sonoda (@mimuret), see + `man 5 dsc.conf` for more details. + + New DNS filters: + - `servfail-only`: Count only SERVFAIL responses + - `authentic-data-only`: Count only DNS messages with the AD bit is set + + New configuration: + - `client_v4_mask`: Set the IPv4 MASK for client_subnet INDEXERS + - `client_v6_mask`: Set the IPv6 MASK for client_subnet INDEXERS + + Fixes: + - Set `_DEFAULT_SOURCE`, was giving compile warnings on some platforms + - Update `pcap-thread` to v2.1.3 for compatibility fixes + - Fix bug where extra `"` would be OK in configuration + - Eat all white-space between tokens in configuration + - Minor documentation corrections + + Commits: + 8a20421 Config parse quote/whitespace bug + 4eb91d8 PR review and corrections + 1dcdbc1 add supports statistics for DNSSEC validation resolver - SERVFAIL + DNS message filter - AD bit DNS message filter - set custom mask + for ClientSubnet + 7c4ce7e Update pcap-thread to v2.1.3 + f5d152c Corrected date + 04f137d Prepare SPEC for OSB/COPR + 402c242 Config header is generated by autotools + +2017-03-29 Jerry Lundström + + Release 2.5.1 + + Various compatibility issues and a possible runtime bug, related to + pcap-thread, fixed. + + Commits: + + 5ed03e3 Compat for OS X + 8605759 Fix compiler warnings + 5fbad26 Update pcap-thread to v2.1.2 + 47ed110 Update pcap-thread to v2.1.1 + +2017-03-02 Jerry Lundström + + Release 2.5.0 + + Resolved memory leaks within the IP fragment reassembly code that was + reported by Klaus Darilion (NIC.AT) and added config option to control + some parts of the fragment handling. + + Fixes: + - Add `pcap_layers_clear_fragments()` to remove old fragments after + `MAX_FRAG_IDLE` (60 seconds) + - Use correct alloc/free functions for dataset hash + - Fix spacing in dsc.conf(5) man-page + + New config option: + - `drop_ip_fragments` will disable IP fragmentation reassembling and + drop any IP packet that is a fragment (even the first) + + Commits: + + eaee6c0 Drop IP fragments + 3ebb687 Issue #146: Fix leak in fragment handling + 9a5e377 Use correct alloc/free + 35f663c Fix #107: add const + +2017-01-27 Jerry Lundström + + Release 2.4.0 + + Since there have been a few major issues with the threaded capturing code + it is now default disabled and have to be enabled with a configure option + to use: `./configure --enable-threads ...` + + A lot of work has been done to ensure stability and correct capturing, + as of now `dsc` is continuously running on the testing platforms with + simulated traffic and tests are performance every 5-15 minutes: + + https://dev.dns-oarc.net/jenkins/view/dsctest/ + + With the rewrite of the config parser to C it was missed that Hapy allowed + CR/LF within the values of the options. Changing the C parser to allow + it is a bit of work and having CR/LF within the value may lead to other + issues so it is now documented that CR/LF are not allowed in config option + values. + + Fixes: + - The `-T` flag was just controlling pcap-thread usage of threads, it now + controls all usage of threads including how signals are caught. + - Fix program name, was incorrectly set so it would be reported as `/dsc`. + - Use thread safe functions (_r). + - Handle very long config lines by not having a static buffer, instead + let `getline()` allocate as needed. + - Use new activation in pcap-thread to activate the capturing of pcaps + after the initial interval sync have been done during start-up. + - Use factions of second for start-up interval sync and interval wait. + - Fix memory leaks if config options was specified more then once. + - Use new absolute timed run in pcap-thread to more exactly end capturing + at the interval. + - Fix config parsing, was checking for tab when should look for line feed. + - Exit correctly during pcap-thread run to honor `dump_reports_on_exit`. + - Use 100ms as default pcap-thread timeout, was 1s before but the old code + used 250ms. + - Various enhancements to logging of errors. + + New config options/features: + - `pcap_buffer_size` can be used to increase the capture buffer within + pcap-thread/libpcap, this can help mitigate dropped packets by the + kernel during interval breaks. + - `no_wait_interval` will skip the interval sync that happens during + start-up and start capturing directly, the end of the interval will + still be the modulus of the interval. + - `pcap_thread_timeout` can be used to change the internal timeout use + in pcap-thread to wait for packets (default 100ms). + - Log non-fatal errors from pcap-thread w.r.t. setting the filter which + can indicate that the filter is running in userland because lack of + support or that it is too large for the kernel. + + Special thanks to: + - Anand Buddhdev, RIPE NCC + - Klaus Darilion, NIC.AT + - Vincent Charrade, Nameshield + + Commits: + + ee59572 Fix #111, fix #116: Update pcap-thread to v2.0.0, remove debug + code + 64befef Update copyright year + 40a1fb4 Fix #139: Use 100ms as default pcap-thread timeout + 2a07185 Fix #137: Graceful exit on signal during run + f1b3ec3 Issue #116: Try and make select issue more clear + 950ea96 Fix #133: Return from `Pcap_run()` on signal/errors + 667cc91 Issue #116: Add config option pcap_thread_timeout + 3c9e073 Notice if non-fatal errors was detected during activation + 4ea8f54 Fix #108: Document that CR/LF are not allowed within configuration + line + 9fda332 Check for LF and not tab + 15a1dc0 Use pcap-thread timed run to interface + 1e98f8b Fix potential memory leaks if config options specified more then + once + a9b38e9 Add missing LF and indicate what config option was wrong if + possible + f8a2821 Use fractions of seconds for both start up interval sync and + timed run, always adjust for inter-run processing delay + f47069a Fix #121: Update to pcap-thread latest develop + fc13d73 Issue #116: Feature for not waiting on the interval sync + c832337 Fix #122: Update pcap-thread to v1.2.3 for fix in timed run + 4739111 Add `pcap_buffer_size` config option + 7d9bf90 Update pcap-thread to v1.2.2 + ef43335 Make threads optional and default disabled + c2399cf getline() returns error on eof, don't report error if we are + 5c671e6 Clarify config error message and report `getline()` error + 8bd6a67 Fix #114: Handle very long lines + 47b1e1a Use _r thread safe functions when possible + 0f5d883 Update daemon.c + f18e3ea Update doc, -T now disables all usage of threads + 57aacbe Honor the -T flag when installing signal handlers + +2016-12-22 Jerry Lundström + + Release 2.3.0 + + Rare lockup has been fixed that could happen if a signal was received + in the wrong thread at the wrong time due to `pcap_thread_stop()` + canceling and waiting on threads to join again. The handling of signals + have been improved for threaded and non-threaded operations. + + A couple of bugfixes, one to fix loading of GeoIP ASN database and + another to use the lowest 32 bits of an IP address (being v4 or v6) + in the IP hash making it a bit more efficient for v6 addresses. + + New functionality for the configure option `local_address`, you can now + specify a network mask (see `man 5 dsc.conf` for syntax). + + Commits: + + e286298 Fix CID 158968 Bad bit shift operation + c15db43 Update to pcap-thread v1.2.1 + 1ac06ac Move stopping process to not require a packet + 597dd34 Handle signals better with and without pthreads + bcf99e8 Add RPM spec and ACLOCAL_AMFLAGS to build on CentOS 6 + 667fe69 fixed load geoIP ASN database from config-file + e1304d4 Fix #97: Add optional mask to `local_address` so you can + specify networks + 5dae7dd Fix #96: Hash the lowest 32 bits of IP addresses + +2016-10-15 Jerry Lundström + + Release 2.2.1 + + Two bugfixes with one being critical (caused segfault, sorry for that) + - `pcap-thread` had an issue that threads where not closed on exit + of `pcap_thread_run()`, this only created many threads in my testing + environment but reports came in that it segfaulted. + - When started `dsc` (not in debug mode) it will wait to align with + the time, that did not get update with the configurable interval + change. + + Commits: + + 6e3654b Fix #90: Update pcap-thread to v1.1.2 to fix a segfault issue + c9350a3 Fix #92: Honor configured interval when aligning + acaf617 minor typo + +2016-10-10 Jerry Lundström + + Release 2.2.0 + + Some big changes in this release are the removal of the C++ configure + parser library Hapy and the addition of pcap-thread to (hopefully) + handle capturing packet in a correct and efficiant way. + + With that addition of pcap-thread comes new runtime options: + - `-m` sets monitor mode on interfaces + - `-i` sets immediate mode on interfaces + - `-T` disable the usage of threads in pcap thread + + Bugfixes: + - d95190a fixes a small memory leak in `Pcap_init()` and a possible + bug where `-p` might never been used because of not being declared + external. + - 55e1056 added check for `netinet/ip_compat.h` for use in + `src/pcap_layers/pcap_layers.c`. + + Commits: + + 0a1ce91 Fix coverity issues + 5a1d410 Delete useless line (related to mayasd#84) + 443db3e Check if the file was previously linked but not yet unlinked + (Tim CLERC.IM) + 02a7621 Fix #82: Oops, `pcap_thread_set_filter()` had changed during + development and missed this + 2a8aa29 Move definition of token struct inside and changed name to not + conflict on FreeBSD + 43da964 Fix #9: Implement conf parser in C and remove dependency of + Hapy and C++ + 9f46f0d Update pcap thread to version 1.1.1 + d95190a Use pcap thread, new options `-miT` and possible bugfix for + `-p` and a small memory leak fix + 55e1056 Fix #77: Check for netinet/ip_compat + 4e120f9 Fix travis script, only expand dir. + +2016-06-28 Jerry Lundström + + Release 2.1.1 + + Bugfixes: + - 22688c1 Fix pcap/select last_ts + In some cases `select` will return the fd set as if there are packets + to read but there aren't. That would case the last timestamp to not + advance and the `Pcap_run` loop to never finish. + This fix add a check on packets captured and sets last timestamp if + none where caught. + + Commits: + + d115b3f Correct configuration, missing `;`. + 22688c1 Fix pcap/select last_ts + b6d3dd8 Fix package dependencies. + c8979c4 Add debian/ubuntu package files + +2016-06-09 Jerry Lundström + + Release 2.1.0 + + This release brings a couple of new features, the ability to change + the interval for which DSC writes out the statistics files and a new + indexer for AS numbers. + There was also an issue detected in Ubuntu 16.04 when reading pcap files + would result in doubling the statistics if running in daemon mode. + Looking at the strace it might be a issue with the kernel, libc, pcap or + a combination that results in the open file handle to reset and essentially + read all the data twice. + + Credits: + - Klaus Darilion NIC.AT for interval changes and AS number indexer. + + Commits: + + 8ab8632 Rewrote ASN indexer to remove the need for malloc/free. Removed + the need to malloc ipstr. Use same unknown tags as country + indexer. + 89d4984 add ASN index: maps src-IP to AS number using GeoIP ASNum DB + 8d5c6bf Fix #14: Remove ncap + c0f00e7 Add test for statistics_interval + cf9ede2 Add checks and documentation for statistics_interval + 93eeecd new config option 'statistics_interval' (defaults to 60s) + a28f5d4 Fix #62: Do not go into daemon mode if reading offline files + 2bc1abb Add information about puppet module + +2016-06-01 Jerry Lundström + + Release 2.0.0 + + This release brings a major update to the DSC software with the separation + of the Collector and the Presenter, this repository will only include + the Collector from now on. + + Please read UPGRADE.md for information on upgrading from previous version. + + Major changes / additions are: + - Use of Automake and rework of the Makefiles + - Conform to FHS 3.0 + - Man-pages (man dsc, man dsc.conf) + - Continuous Integration testing using Travis-CI + - Compatibility testing on Debian, CentOS, FreeBSD and OpenBSD + - Use of Coverity Scan to find defects + - JSON output format, see output_format in dsc.conf(5). + - IPv6 support in country indexer and libgeoip is now runtime + configurable, see geoip_v4_dat / geoip_v6_dat in dsc.conf(5). + - Signal handling and optional write reports on exit, see + dump_reports_on_exit in dsc.conf(5). + - Upload scripts are deprecated + + Credits: + - Klaus Darilion NIC.AT for GeoIP IPv6 patch. + - Michael Braunoeder NIC.AT for NXDOMAIN filter patch. + - L-root for overflow bugfix IP fragments. + - McStork for JSON output patch. + + Changes since release candidate: + + 1be5148 Fix #57: Flush the pid file to write it out and add test for + pid file + 0f79aa0 Use Semantic Versioning 2.0.0 semver.org + +2016-05-10 Jerry Lundström + + Release 2.0.0-rc.1 + + 69ef9b4 Add -v to display version + 7e5b403 Fix defects + 0f64128 Add badges + f795ed3 Old automake needs AM_PROG_CC_C_O + 27ae870 Fix #4: Remove old indexers, update indexers and filters + documentation, update authors. + d873411 Fix #38: Use locking to ensure we do not overwrite PID file + 800fe83 Fix #35: GeoIP configurable. Fix debug and syslog in country + indexer. Add documentation and config example. + a1dcdf1 Avoid the checks for newer .dat files, this may have performance + impacts. + bb9a059 Add IPv6 support to country indexer and add some failure handling + 03e16a8 Include config.h as this defines HAVE_LIBGEOIP. Thus, the geoip + code was not used yet. + 7808d2e New dsyslog/dsyslogf/dfprint macro. Change fprintf to dfprintf. + fe47288 Fix #34: Create man-pages. + 7a9b3c3 Fix #33: Handle most signals, new config option to dump reports + on exit. + 4753eda Add a filter to track NXDOMAIN responses + 1565952 Add info about DSP + 05ef699 Fix make test in dist. + 68bdc9b No default interface in conf example + 2be98be Correct libexec directory. Install etc files in subdirectory. + 319ac4e Only build dist. Build in a build directory. + 96e0e73 Fix #16: conform to FHS. Automake tweaks. + bbcca74 Fix #23: Add missing changes for master branch + 197ad52 No need to install autoconf/automake in Travis + eb95ee1 Add license to cron scripts + 4f62420 Fix/add make dist and try it in Travis + ac4c634 Reconstruct repository to move out presenter. Update licenses. + Use pcap_layers as a git submodule. Use automake/autoconf. + 89c7f4c Import patched pcap_layers.c code with buffer overflow bugfix + 2787db2 Only wait a certain number of times for the files to appear + b5d911d Add the first test, simple run and compare gold files + 32fd807 Fix #13: Document ability to read packets from pcap files + eee217e Rework some of the Makefiles based on some of the patches found in + the Debian packages + 7a2a67e Fix #12: Add hash for dataset names and check for duplicates + ed1eba9 Add base64 for certain non-printable characters in JSON output and + remove extra new-line + 580d543 Add output_format and JSON structure description to documentation + d2d1ed2 Fix #3: Reworked JSON output format implementation + 2e2f90f Give option for additional output JSON/Extended JSON + a23b6af Ignore generated files + f4214f3 Add Travis CI. Remove old TODAY/tar commands. Add ifndef/def to + all .h files. Fix issue with arpa/nameser_compat.h on OpenBSD by + checking for the header file and only including it if it exists. + Rename configure.scan to configure.ac and change contact info and + version. Update configure using autoconf 2.69 on Ubuntu 14.04 LTS. + Set CC/CPP/CXX/CXXFLAGS in Makefile found by configure. + +2016-01-11 Duane Wessels + + added dfprintf() macro to improve code readability and avoid + multi-line if (debug_lvl) fprintf(...) statements. + +2016-01-11 Duane Wessels + + Commenting out ancount and nscount to silence compiler warnings + about unused variables. + +2016-01-11 Duane Wessels + + Patch from John Heidemann relating to TCP reassembly + - fix for multiple DNS messages per TCP connection that span + multiple segments + - some editorial code changes (== to >=) + - additional debugging statements + +2016-01-11 Duane Wessels + + Minor fix: debugging statement wasn't protected with debug_flag + check. (thanks John H). + +2015-12-23 Duane Wessels + + add prototype for dns_message_handle() + +2015-12-23 Duane Wessels + + Experimental feature to drop "received responses" and "sent queries" + since DSC is generally used to monitor authoritative servers that, + under normal operation, never receive responses nor send queries. + Currently protected with #ifdefs. + +2015-12-15 Duane Wessels + + Oops, forgot to set dns_message->server_ip_addr for new server_addr + indexer + +2015-11-23 Duane Wessels + + removing one level of the "dsc/dsc" top directory + +2015-11-11 aqadeer + + In pcap.c pcap_setnonblock doesn't accept captured files anymore and + must need a device from where it could do live capture. For offline + files to work, a simple check is added to by-pass this porblem. + +2015-11-05 Duane Wessels + + Added a 'server' indexer. This records the server (query destination) + IP address. + +2015-11-05 Duane Wessels + + Renamed "client_ipv4" to "client_ip" because it supports v4 and v6 + +2015-11-04 Duane Wessels + + Update copy of pcap_layers.c from https://github.com/wessels/pcap_layers + +2015-11-04 Duane Wessels + + avoid "void *" pointer arithmetic (Yoshitaka Aharen) + +2015-10-09 Duane Wessels + + bugfix: handle receiving TCP dns length prefix out of order. + + A user reported that when a TCP segment containing only the DNS message + length is received *after* the message it references (i.e., out-of-order), + then DSC goes into a 100% CPU loop. Confirmed that the code doesn't work + correctly when dnslen comes out-of-order, but I wasn't able to easily + reproduce the 100% CPU bug. + + The fix is to add a "dnslen_bytes_seen_mask" variable that tracks which + of the two dnslen bytes we've seen. Once both bytes have been seen, + then we can proceed to reassembling the message buffer. + +2015-10-09 Duane Wessels + + cosmetic: rename ipv4 to ip4 to be consistent with ip6 + +2015-10-09 Duane Wessels + + Regarding recent change to add pcap_layers library, forgot to add it + to Makefile.in (vs Makefile) before committing. + +2015-10-09 Duane Wessels + + Integrated https://github.com/wessels/pcap_layers with DSC collector. + The DSC code now includes a copy of the pcap_layers library, which does + a better job of extracting the layers of a pcap packet. In particular, + it does IP fragmentation reassembly, which is important for DSC and + RSSAC-002. At this time the DSC code still does TCP reassembly, however. + + Removed USE_IPV6 ifdefs. IPV6 support is now always compiled. + +2015-10-08 Duane Wessels + + Removing "DMC *dns_message_callback" because there is only one that + would ever be used. The callback layer of indirection makes the code + a little confusing, and also this change is in preparation for bringing + in a third-party pcap layer handling library which will work slightly + differently. + +2015-10-08 Duane Wessels + + The "ip_message" code has not been in use since a commit back in 2012. + But the code lingered, until now. + +2015-02-25 Duane Wessels + + User reported a concern with the way dsc-xml-extractor.pl called + the operating system 'mv' command. It has been replaced with Perl's + File::Copy::mv(). + +2013-03-19 Duane Wessels + + Fixed a bug in TCP reassembly when the DNS length field was + split between two segments. + +2012-08-27 Sebastian Castro + + Added 'tc_bit' indexer and dataset to track the frequency + of responses having the TC bit set. Useful with DNSSEC + signed zones. + +2012-02-29 Duane Wessels + + Added 'qr_aa_bits' dataset and graph. It shows the distribution + of QR/AA values in received messages and may be helpful in + detecting reflector attacks targeting your name server. + +2010-11-29 Duane Wessels + + The dnssec_qtype and dns_ip_version plots were not working due + the presence of 'dataset' attributes in their definitions. Also + fixed the chaos plot. + +2010-11-04 Duane Wessels + + Added NSEC3 record type to the extractor. + +2010-09-14 Duane Wessels + + A fix to put pcap interfaces into non-blocking mode, which is + important when reading from multiple interfaces. + +2010-08-13 Alexander Mayrhofer + + Added "country_index" to the collector which is an index based + on country as returned by the GeoIP library. configure attempts + to learn if you have GeoIP installed. + +2010-08-13 Peter Koch + + Some minor fixes to the collector that allow it to read from + pcap files on disk, rather than live packets from an interface. + +2010-06-02 Henrik Kramsh?j + + Improvements for OpenBSD compatibility + +2010-05-01 Duane Wessels + + Added NSEC3 to DNSSEC query types + +2010-02-22 Duane Wessels + + Collector bug fixed: USE_IPV6=1 was not passed to .c + files, which resulted in different-sized data structurs + and crashes on CentOS. + +2009-12-10 Duane Wessels + + Added "priming_queries" and "priming_responses" datasets in + preparation for root zone signing. + +2009-11-11 Duane Wessels + + On the presenter some of the "accum" graphs stopped working + due to recent rewrites. These should now be working again. + +2009-10-12 Duane Wessels + + The presenter debugging is now configurable via dsc-grapher.cfg. + +2009-08-14 Duane Wessels + + In the presenter, there have been some significant changes to + the perl modules so that they can be used in command line + (ie non-CGI) utilities, both for creating graphs and for + reading data. Most of these changes are purely internal. + However, you may need to update your dsc-grapher.pl CGI + program to be like presenter/grapher/dsc-grapher.pl.sample + in the source distribution. + +2009-04-15 Duane Wessels + + In the presenter's refile-and-grok.sh script, note that in + some environtments it is useful to skip NODEs that don't + have an incoming directory because they might have been + "grokked" elsewhere and then rsync'd to you. + +2009-02-27 Duane Wessels + + In the collector it is now necessary to include <stdlib.h> + and <stdio.h>. Not sure why it wasn't necessary before. + Perhaps due to 2009-01-26 Hapy library upgrade. + +2009-02-19 Duane Wessels + + In the collector, fixed some 64-bit free disk space calculations + with casting. + +2009-01-26 Duane Wessels + + Upgraded to more recent version of Hapy parsing library. + +2008-12-31 Duane Wessels + + In the collector, changed the value of pcap_open_live() because + we always try to read from all interfaces after select() returns. + If we have multiple interfaces and one of them is quiet, and + the timeout is too large, we'll drop packets on the other + interface while waiting for the first to time out. + + Also added a pseudo-dataset that reports on pcap packets captured + and dropped. + + +2008-12-02 Duane Wessels + + Grapher now preserves the order of the server list given in + dsc-grapher.cfg. + +2008-11-22 Duane Wessels + + Added some collector indexers and datasets for 2nd and 3rd level + domains. + +2008-10-02 Duane Wessels + + Made the 'no extractor for $dataset' error message non-fatal. + Now, the unknown dataset will just be skipped and the remaining + ones will be processed. + +2008-09-30 Duane Wessels + + In the presenter, account for the possibility that the collector's + clock is not in sync. + +2008-09-18 Elmar Knipp + + Found bug in upload script where a $PROG.out prevents removal + of a $YYYYMMDD directory. + +2008-09-16 Duane Wessels + + New presenter feature: Node Merging. Now the DSC grapher can + automatically "merge" multiple nodes so that they appear as + one. For example instead of this dsc-grapher.cfg line: + + server S N1a N1b + + you can merge nodes "a" and "b" with this line: + + server S N1=N1a,N1b + + This feature is also useful for just renaming a node if you + want it to be displayed as a different name than the directory + where its files are. + +2008-09-15 Duane Wessels + + On the collector, changed the sample config and upload scripts + to use /usr/local/dsc/run as the default run_dir (was formerly + /usr/local/dsc/var/run/). + +2008-08-21 Duane Wessels + + Added an EDNS bufsiz indexer so we can collect buffer sizes + advertised by clients. + +2008-08-12 Duane Wessels + + Attempt to improve portability (on Solaris) by checking for + libresolv, libnsl, and libsocket. Also check for statvfs() vs + statvs() in configure. + +2008-07-29 Duane Wessels + + Updates to collector/cron/upload-rsync.sh so that it behaves + better with new date-based directories on both collector and + presenter. + +2008-06-27 Duane Wessels + + Added a source port range dataset (for about-to-be-announced + vulnerability that can be mititgated by improving source port + randomness). + +2008-04-22 Duane Wessels + + Added ./configure script for presenter/dsc code. + +2008-04-22 Duane Wessels + + Added support for capturing NCAP data streams. + +2008-01-09 Duane Wessels + + Changed the location of the presenter's ".dat" files. Previously + there were stored in a directory such as 20080109/qtype/qtype.dat + and now they will be moved one directory up and stored as + 20080109/qtype.dat. + + The source distribution includes a script found at + presenter/grapher/update-dat-file-locations.sh that will traverse + the /usr/local/dsc/data directory and move all the .dat files + one level up. + + When upgrading to this version of DSC you should first stop the + refile-and-grok.sh cron job, install the DSC software, and then + run the update-dat-file-locations.sh script (after reading it). + +2008-01-09 Duane Wessels + + Data passed from collector to presenter is now contained in a + single XML file, rather than one XML file per dataset. This + should significantly reduce filesystem pressure (i.e., 20x fewer + files to be stored and sent) on both sides. + +2008-01-07 Duane Wessels + + The collector now stores to-be-uploaded XML files in + date-named subdirectories. This reduces filesystem pressure + in the event that the collector cannot communicate with the + presenter for an extended period of time. + + To accomodate this change, the upload-prep.sh script has been + replaced with a Perl version, named upload-prep.pl. After + installing DSC you should change your cron job to use the new + script. + +2008-01-04 Ken Keys + + Previously, a child was forked for each collection interval. + Memory allocated for collected data was freed by simply exiting + the child. But this left no way to preserve any data across + intervals. + + Now, interval collectors do not fork. To avoid the tedious, + slow, and error-prone nature of normal malloc-and-free memory + management, we instead store non-persistent data in memory + allocated from an "arena", which can be freed quickly and safely. + We don't need to free individual allocations, but we do need + to reset some pointers to allocated memory. + + TCP resets are now properly handled and TCP state is freed + if a connection has been idle for too long. + +2008-01-04 Duane Wessels + + The collector configuration (dsc.conf) has a new 'minfree_bytes' + directive. If the amount of free disk space on dsc's partition + falls below this limit, dsc will not write any XML files -- that + data will be lost. + + The default value is 5 MB. + +2007-12-14 Alexander Gall + + Added a 'dns_ip_version' indexer so that we can track DNS messages + sent over v4 and v6. + +2007-12-04 Duane Wessels + + Fixed a collector bug when listening on multiple interfaces of + different datalink types (e.g., eth0 and lo). + +2007-11-26 Duane Wessels + + Added a 'hide_nodes' feature to dsc-grapher.cfg. If 'hide_nodes' + is given, then the navigation menu will not display the nodes + underneath a server. Note, however, that knowledgeable users + could still view individual node data by manually if they know + the names by specifying it in the URL query terms. + +2007-10-09 Duane Wessels + + Changed the presenter to not use "data URIs" by default. Now + Internet Explorer users should be happier. + +2007/06/14 Duane Wessels + + Added -f command line option to keep 'dsc' collector as a foreground + process. + +2007-06-11 Duane Wessels + + dsc-xml-extractor.pl now looks for incoming XML files in + "incoming/YYYY-MM-DD" directories. + +2007-06-07 Duane Wessels + + New TCP code did not compile cleanly on Linux. Added some + #ifdefs and macros for Linux compatibility on TCP headers. + +2007-05-15 Ken Keys, Duane Wessels + + TCP support + New 'transport' indexer for DNS messages. Indicates whether message + arrived over UDP or TCP. + +2007-04-04 Duane Wessels + + IPv6 support + New 'ip_version' indexer. Indicates whether DNS message arrived + via IPv4 or IPv6. |