Everyone keeps data. Big organizations spend millions to look after their payroll, customer and transaction data. The penalties for getting it wrong are severe: businesses may collapse, shareholders and customers lose money, and for many organizations (airlines, health boards, energy companies), it is not exaggerating to say that even personal safety may be put at risk. And then there are the lawsuits. The problems in successfully designing, installing, and maintaining such large databases are the subject of numerous books on data management and software engineering. However, many small databases are used within large organizations and also for small businesses, clubs, and private concerns. When these go wrong, it doesn't make the front page of the papers; but the costs, often hidden, can be just as serious.
Where do we find these smaller electronic databases? Sports clubs will have membership information and match results; small businesses might maintain their own customer data. Within large organizations, there will also be a number of small projects to maintain data information that isn't easily or conveniently managed by the large system-wide databases. Researchers may keep their own experiment and survey results; groups will want to manage their own rosters or keep track of equipment; departments may keep their own detailed accounts and submit just a summary to the organization's financial software.
Most of these small databases are set up by end users. These are people whose main job is something other than that of a Computer professional. They will typically be scientists, administrators, technicians, accountants, or teachers, and many will have only modest skills when it comes to spreadsheet or database software.
The resulting databases often do not live up to expectations. Time and energy is expended to set up a few tables in a database product such as Microsoft Access, or in setting up a spreadsheet in a product such as Excel. Even more time is spent collecting and keying in data. But invariably (often within a short time frame) there is a problem producing what seems to be a quite simple report or query. Often this is because the way the tables have been set up makes the required result very awkward, if not impossible, to achieve.
A database that does not fulfill expectations becomes a
costly exercise in more ways than one. We clearly have the
cost of the time and effort expended on setting up an
unsatisfactory application. However, a much more serious
problem is the unability to make the best use of valuable
data. This is especially so for research data. Scientific and
social researchers may spend considerable money and many
years designing experiments, hiring assistants and collecting
and analyzing data, but often very little thought goes into
storing it in an appropriately designed database.
Unfortunately, some quite simple mistakes in design can mean that much of the potential information is lost. The
immediate objective may be satisfied, but unforeseen uses
of the data may be seriously compromised. Next year's grant
opportunities are lost.
The word awkward, in the fourth paragraph: