Performance data processing jobs and massive data sets in SQL
by Mountain Computers Inc, Publication Date: Saturday, May 4, 2024
View Count: 329, Keywords: SQL, Distributed processing, Data Processing, Hashtags: #SQL #Distributedprocessing #DataProcessing
In the midst of cloud computing, on premise data processing of large datasets is still quite popular. Of course my choice of data storage is SQL, yet I might have to deviate and choose another storage format.
For example, the data sets I would download from Open Data was immense and the processing of 800 million to 1 billion rows of records for whatever intensive purposes was even more impressive on premise in my lab.
Today, I am in the midst of an architectural design challenge to collect massive data via software agents abroad and securely receive, store, and process. What is challenge? The secure receiving part of the data transfer. yes, there are vendors that have various solutions, yet what about open source solutions that are secure and robust.
That is my quandary. Give me a few more weeks of thought and I am sure an epiphany will occur.
Till then, happy data processing.
more to come...
if you found this article helpful, consider contributing $10, 20 an Andrew Jackson or so..to the author. more authors coming soon
FYI we use paypal or patreon, patreon has 3x the transaction fees, so we don't, not yet.
© 2025 myBlog™ v1.1 All rights reserved. We count views as reads, so let's not over think it.
