Data Engineering

All things related to data engineering - databases, distributed systems, big data and much more

Sources

Updated at
Fri May 7 16:18:01

	Windows task scheduler? by /u/Limp-Ad-7289 19 minutes ago.
	Certs needed by /u/pleasecometalktome 37 minutes ago.
	What are some good resume projects for a new grad? by /u/the_originalposter 1 hour ago.
	How Apache Flink manages Kafka consumer offsets by /u/Marksfik 2 hours ago.
	Apache Flink SQL client on Docker by /u/ftisiot 3 hours ago.
	Configure Airflow 2.0 with LDAP in 15 mins by /u/marclamberti 4 hours ago.
	Suggested courses / material / books to start working as an DE by /u/D1yzz 5 hours ago.
	Under the Hood of an Exadata Transaction – How We Harnessed the Power of Persistent Memory (Jia Shi) by CMU Database Group 6 hours ago.
	Scheduling tools for ETL and ML flow by /u/r20367585 6 hours ago.
	Comparing ideas around a DB use case. by /u/a-r-t-u-r 7 hours ago.
	Data loading from a SQL database to a warehouse by /u/owila 10 hours ago.
	Master vs Bachelor degree + experience by /u/adpascual 12 hours ago.
	What open source Business Intelligence tool are you or your team using? by /u/seanbayarea 13 hours ago.
	Making Spark Cloud Native At Data Mechanics - Episode 184 by tmacey 14 hours ago.
	Yayy! I finished my second project but I would like feed back, my data pipeline is blood slow! by /u/YaswanthBangaru 16 hours ago.
	Request for Advice - DE role with engineering background by /u/Suitable_Space_1104 18 hours ago.
	Data Quality tool by /u/soujoshi 20 hours ago.
	Optimal Feature Discovery: Better, Leaner Machine Learning Models Through Information Theory by Adam Wang 23 hours ago.
	The Grand Vision And Present Reality of DataOps - Episode 183 by tmacey 3 days ago.
	SingleStore: Separation of Storage and Compute for Transactions and Analytics (Joyo Victor) by CMU Database Group 5 days ago.
	Automating Merchant Live Monitoring with Real-Time Analytics: Charon by Marco Vita 7 days ago.
	eXtremeDB: Deterministic Database Management in Mission-Critical Applications (Andrei Gorine) by CMU Database Group 8 days ago.
	Freight Pricing with a Controlled Markov Decision Process by Guillaume De Roo 10 days ago.
	Self Service Data Exploration And Dashboarding With Superset - Episode 182 by tmacey 10 days ago.
	LeanStore: In-Memory Data Management Beyond Main Memory (Viktor Leis) by CMU Database Group 17 days ago.
	Moving Machine Learning Into The Data Pipeline at Cherre - Episode 181 by tmacey 17 days ago.
	Query Processing in Google BigQuery (Hossein Ahmadi + Aleksandras Surna) by CMU Database Group 17 days ago.
	FASTER: Efficient State Management for the Modern Edge-Cloud (Badrish Chandramouli) by CMU Database Group 20 days ago.
	Exploring The Expanding Landscape Of Data Professions with Josh Benamram of Databand - Episode 180 by tmacey 24 days ago.
	Flipr: Making Changes Quickly and Safely at Scale by Andy Maule 24 days ago.
	Put Your Whole Data Team On The Same Page With Atlan - Episode 179 by tmacey 31 days ago.
	Data Quality Management For The Whole Team With Soda Data - Episode 178 by tmacey 38 days ago.
	NoisePage: The Self-Driving Database Management System (Lin Ma) by CMU Database Group 42 days ago.
	My thoughts on the data mesh by Daniel Abadi 43 days ago.
	Real World Change Data Capture At Datacoral - Episode 177 by tmacey 45 days ago.
	HarperDB’s Data Storage Journey: From File System to LMDB (Kyle Bernhardy) by CMU Database Group 47 days ago.
	Celebration of Mike Stonebraker #6 - MIT Impact (Sam Madden) by CMU Database Group 50 days ago.
	Celebration of Mike Stonebraker #8 - Students (Magda Balazinska, Andy Pavlo) by CMU Database Group 50 days ago.
	Celebration of Mike Stonebraker #10 - Closing Remarks (Mike Stonebraker) by CMU Database Group 50 days ago.
	Uber’s Journey Toward Better Data Culture From First Principles by Krishna Puttaswamy 52 days ago.
	Managing The DoorDash Data Platform - Episode 176 by tmacey 52 days ago.
	Celebration of Mike Stonebraker #5 - Streaming Databases (Stan Zdonik) by CMU Database Group 53 days ago.
	Celebration of Mike Stonebraker #2 - 44 Years of Mike (David DeWitt) by CMU Database Group 55 days ago.
	Celebration of Mike Stonebraker #7 - Vertica & Column Stores (Andy Palmer) by CMU Database Group 55 days ago.
	Celebration of Mike Stonebraker #9 - Open Mic + Messages by CMU Database Group 55 days ago.
	Navigating to the Technical Program Management and Learning Team 55 days ago.
	Leave Your Data Where It Is And Automate Feature Extraction With Molecula - Episode 175 by tmacey 59 days ago.
	Elastic Deep Learning with Horovod on Ray by Travis Addair 59 days ago.
	Applying Machine Learning in Internal Audit with Sparsely Labeled Data by Jesse He 65 days ago.
	How Uber Deals with Large iOS App Size by Milind Chabbi 69 days ago.
	Evolving Schemaless into a Distributed SQL Database by Ovais Tariq 72 days ago.
	Fast and Reliable Schema-Agnostic Log Analytics Platform by Chao Wang 76 days ago.
	Uber’s Real-time Data Intelligence Platform At Scale: Improving Gairos Scalability/Reliability by Gang Zhao 107 days ago.
	The Journey Towards Metric Standardization by Xiaodong Wang 114 days ago.
	Disaster Recovery for Multi-Region Kafka at Uber by Yupeng Fu 136 days ago.
	Uber’s Real-Time Push Platform by Uday Kiran Medisetty 139 days ago.
	No Code Workflow Orchestrator for Building Batch & Streaming Pipelines at Scale by Sriharsha Chintalapani 146 days ago.
	Horovod v0.21: Optimizing Network Utilization with Local Gradient Aggregation and Grouped Allreduce by Travis Addair 163 days ago.
	Turning Metadata Into Insights with Databook by Sunheng Taing 177 days ago.
	Meet the 2020 Safety Engineering Interns: COVID Edition by Safety Engineering Interns 190 days ago.
	Operating Apache Pinot @ Uber Scale by Yupeng Fu 199 days ago.
	Building from the Baltics: Meet the Uber Engineering Team in Vilnius, Lithuania by Ieva Sapokaite 204 days ago.
	Ludwig v0.3 Introduces Hyperparameter Optimization, Transformers and TensorFlow 2 support by Travis Addair 213 days ago.
	Revolutionizing Money Movements at Scale with Strong Data Consistency by Aakriti Singla 217 days ago.
	Spearheading Open Source: A Conversation with Jim Jagielski, Staff Technical Program Manager with the Uber Open Source Program Office by Uber Open Source Office 225 days ago.
	Designing Edge Gateway, Uber’s API Lifecycle Management Platform by Madan Thangavelu 262 days ago.
	Standing for Safety: Meet the Uber Sao Paulo Tech Team by Camila Carvalho 283 days ago.
	Introducing Domain-Oriented Microservice Architecture by Adam Gluck 288 days ago.
	Engineering Failover Handling in Uber’s Mobile Networking Infrastructure by Sivabalan Narayanan 297 days ago.
	Fiber: Distributed Computing for AI Made Simple by Jiale Zhi 311 days ago.
	Editing Massive Geospatial Data Sets with nebula.gl by Georgios Karnas 318 days ago.
	Profiles in Coding: Diana Yanakiev, Uber ATG, Pittsburgh by Bea Schuster 325 days ago.
	Building a Large-scale Transactional Data Lake at Uber Using Apache Hudi by Nishith Agarwal 332 days ago.
	Introducing Neuropod, Uber ATG’s Open Source Deep Learning Inference Engine by Vivek Panyam 333 days ago.
	Inside Uber ATG’s Data Mining Operation: Identifying Real Road Scenarios at Scale for Machine Learning by Steffon Davis 339 days ago.
	Meta-Graph: Few-Shot Link Prediction Using Meta-Learning by Ankit Jain 343 days ago.
	Profiles in Coding: Christabelle Bosson, Uber Elevate, San Francisco by Bea Schuster 344 days ago.
	Developing the Next Generation of Coders with the Dev/Mission Uber Coding Fellowship by Kurtis Nusbaum 350 days ago.
	Introducing Athenadriver: An Open Source Amazon Athena Database Driver for Go by Henry Fuheng Wu 352 days ago.
	Building Uber’s Go Monorepo with Bazel by Zhongpeng Lin 358 days ago.
	Announcing a New Framework for Designing Optimal Experiments with Pyro by Adam Foster 360 days ago.
	It's time to rethink how we share data on the Web by Daniel Abadi 505 days ago.
	Introducing SLOG: Cheating the low-latency vs. strict serializability tradeoff by Daniel Abadi 518 days ago.
	The dangers of conditional consistency guarantees by Daniel Abadi 518 days ago.
	Introduction to Transaction Isolation Levels by Daniel Abadi 518 days ago.
	Correctness Anomalies Under Serializable Isolation by Daniel Abadi 518 days ago.
	Overview of Consistency Levels in Database Systems by Daniel Abadi 518 days ago.
	An explanation of the difference between Isolation levels vs. Consistency levels by Daniel Abadi 518 days ago.
	Partitioned consensus and its impact on Spanner’s latency by Daniel Abadi 519 days ago.
	It’s Time to Move on from Two Phase Commit by Daniel Abadi 519 days ago.
	NewSQL database systems are failing to guarantee consistency, and I blame Spanner by Daniel Abadi 519 days ago.
	An analysis of the strengths and weaknesses of Apache Arrow by Daniel Abadi 1137 days ago.
	Apache Arrow vs. Parquet and ORC: Do we really need a third Apache project for columnar data representation? by Daniel Abadi 1284 days ago.
	Hazelcast and the Mythical PA/EC System by Daniel Abadi 1306 days ago.
	Scala Days 2017 - Jepsen Keynote by Aphyr 1464 days ago.
	Distributed consistency at scale: Spanner vs. Calvin by Daniel Abadi 1491 days ago.
	Jepsen: CockroachDB beta-20160829 by Aphyr 1541 days ago.
	Jepsen: MongoDB 3.4.0-rc3 by Aphyr 1550 days ago.
	IEEE Computer issue on the CAP Theorem by Daniel Abadi 1726 days ago.
	Jepsen: VoltDB 6.3 by Aphyr 1759 days ago.
	Jepsen: Crate 0.54.9 version divergence by Aphyr 1773 days ago.
	Jepsen: RethinkDB 2.2.3 reconfiguration by Aphyr 1931 days ago.
	Jepsen: RethinkDB 2.1.5 by Aphyr 1950 days ago.
	Why MongoDB, Cassandra, HBase, DynamoDB, and Riak will only let you perform transactions on a single data item by Daniel Abadi 2015 days ago.
	Jepsen: Percona XtraDB Cluster by Aphyr 2071 days ago.
	Jepsen: MariaDB Galera Cluster by Aphyr 2074 days ago.
	Jepsen: Chronos by Aphyr 2096 days ago.
	Jepsen: Aerospike by Aphyr 2194 days ago.
	Jepsen: Elasticsearch 1.5.0 by Aphyr 2201 days ago.
	Jepsen: MongoDB stale reads by Aphyr 2208 days ago.
	Jepsen: Elasticsearch by Aphyr 2517 days ago.
	Jepsen: etcd and Consul by Aphyr 2523 days ago.
	Jepsen: RabbitMQ by Aphyr 2527 days ago.
	Computational techniques in Knossos by Aphyr 2546 days ago.
	Strong consistency models by Aphyr 2549 days ago.
	Jepsen: Redis redux by Aphyr 2704 days ago.
	Jepsen: Strangeloop Hangout by Aphyr 2777 days ago.
	Jepsen: RICON East talk by Aphyr 2777 days ago.
	Jepsen: Cassandra by Aphyr 2781 days ago.
	Jepsen: Kafka by Aphyr 2782 days ago.
	Jepsen: NuoDB by Aphyr 2783 days ago.
	Jepsen: Zookeeper by Aphyr 2783 days ago.
	Automating Jepsen by Aphyr 2869 days ago.
	The network is reliable by Aphyr 2895 days ago.
	Asynchronous replication with failover by Aphyr 2907 days ago.
	Jepsen: final thoughts by Aphyr 2909 days ago.
	Jepsen: Riak by Aphyr 2909 days ago.
	Jepsen: MongoDB by Aphyr 2910 days ago.
	Jepsen: On the perils of network partitions by Aphyr 2910 days ago.
	Jepsen: Redis by Aphyr 2910 days ago.
	Jepsen: Postgres by Aphyr 2911 days ago.
	Is Upstart the right way to get college student start-ups funded? by Daniel Abadi 3103 days ago.
	Defending Matt Welsh’s 'Startup University' Post by Daniel Abadi 3237 days ago.
	If all these new DBMS technologies are so scalable, why are Oracle and DB2 still on top of TPC-C? A roadmap to end their dominance. by Daniel Abadi 3277 days ago.
	Replication and the latency-consistency tradeoff by Daniel Abadi 3438 days ago.
	Overview of the Oracle NoSQL Database by Daniel Abadi 3499 days ago.
	Hadoop's tremendous inefficiency on graph data management (and how to avoid it) by Daniel Abadi 3570 days ago.
	Why Sam Madden is wrong about peer review by Daniel Abadi 3641 days ago.
	Why I'm doing a start-up pre-tenure by Daniel Abadi 3692 days ago.

Data Engineering

All things related to data engineering - databases, distributed systems, big data and much more

Sources

Updated at
Fri May 7 16:18:01