Campbell Ritchie wrote:Welcome to the Ranch
I presume you are already familiar with Java‑database connections? I am not convinced you can stream real‑time data from a database. It takes time to enter data into the database, you would probably want to do that via a transaction which must complete first, an network connections will slow things down, so who knows how long it will take for the data to go from source to your app. It could be a second or even more.
What would happen if you have a trigger on some of your data to run a query whenever you update something?
Moving to our databases forum.
The secret of how to be miserable is to constantly expect things are going to happen the way that they are "supposed" to happen.
You can have faith, which carries the understanding that you may be disappointed. Then there's being a willfully-blind idiot, which virtually guarantees it.
l need how to stream, in realtime, the changes made to a table in a Microsoft SQL server database to a java application
Ron McLeod wrote:
l need how to stream, in realtime, the changes made to a table in a Microsoft SQL server database to a java application
I looked at this recently - I wanted to get a stream of database changes which I could feed it in to Kafka to do some real-time analysis.
The two approaches that I read about are to either query the database tables periodically and look for changes (may require an additional field such as a sequnece number or timestamp to identify when a record has been inserted/changed), or to watch the database's redo log (change log / commit log / journal) and use the extract the change events (similar to how one database would synchronize/replicate with another).
The first approach is probably fine for smaller data sets, but unlikely to scale well when dealing with a large number of records. The second approach requires you to understand how the structure of redo log how to interpret the data - probably not an easy task.
A third approach, which would require changes to the code, would be to enhance the data layer so that in addition to making changes to the database, that it would also generate the event information you want.
The secret of how to be miserable is to constantly expect things are going to happen the way that they are "supposed" to happen.
You can have faith, which carries the understanding that you may be disappointed. Then there's being a willfully-blind idiot, which virtually guarantees it.
It's good to be able to use someting, it's better to understand how it works.
www.goanation.net
It's good to be able to use someting, it's better to understand how it works.
www.goanation.net
Daniel Demesmaecker wrote:I guess my answer is
mutemoot,
The secret of how to be miserable is to constantly expect things are going to happen the way that they are "supposed" to happen.
You can have faith, which carries the understanding that you may be disappointed. Then there's being a willfully-blind idiot, which virtually guarantees it.
Consider Paul's rocket mass heater. |