​Re: Preventing Duplicate Data WAS UUID vs Longint primary key

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

​Re: Preventing Duplicate Data WAS UUID vs Longint primary key

4D Tech mailing list
On Mon, Aug 7, 2017 at 3:00 PM,


David Adams <[hidden email]> wrote:

>
>
> > How do you deal with that problem (Preventing duplicate data)
>

Definitely

"Carefully program your system to detect and prevent duplicate rows" as
appropriate. Generally such a Dupe Check can take many forms depending on
the business needs, the data in question, and the data entry
process/environment. I've not yet found a solution that fits all. Generally
it is much easier to do this if the "data entry user" is a browser post -
where you have the time and space to do more complicated look-ups. (More
and more of my own projects are web front ends to 4D in the backend.)
 ​I've used similar constructions as John's ContactsDuplicateManager example
as well, although I steer away from storing extra data if I can. For less
immediacy data needs I've found that after hours "helper" routines that
fire off and run in the background working through data to flag dupes for
admin oversight next day is popular with some managers who prefer to make
their own decision about whether some stuff really is a dupe or not. Some
duplicate data has to be eliminated "right now" before the record is saved;
whereas some might be interesting to investigate a little more in depth.
The whys and wherefores --what caused it, who caused it, where did it come
from, and why, is it legit or a real mistake; such questions can often lead
to better processes, better training, better form design, better import
provisioning, or better import pre-cleaning, etc. It all depends on the
project and biz needs.
____________
Steve Simpson
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ​Re: Preventing Duplicate Data WAS UUID vs Longint primary key

4D Tech mailing list
I've found that after hours "helper" routines that fire off and run in the
> background working through data to flag dupes for admin oversight next
day is
> popular with some managers who prefer to make their own decision about
whether
> some stuff really is a dupe or not. Some duplicate data has to be
eliminated
> "right now" before the record is saved; whereas some might be interesting
to
> investigate a little more in depth. The whys and wherefores --what caused
it,
> who caused it, where did it come from, and why, is it legit or a real
mistake;
> such questions can often lead to better processes, better training, better
> form design, better import provisioning, or better import pre-cleaning,
etc.
> It all depends on the project and biz needs.

Nicely described. Human judgement is pretty important or critical with some
"duplicate" row choices. Says the guy with a name so common he's seen his
name taken of a flight manifest because they already had one David Adams on
board. Man, that guy sucks. Absolutely agree about the training benefits of
*quick* feedback on duplicates. You run a scan at night, find a list of
possible duplicates and then can go and find the person responsible. Like
you said, the perfect opportuity for better data entry tools ("Oh, the big
screen is too hard to enter when we get a phone call, so we always just add
a new customr record"), better training (see previous) and, hopefully,
better communications amongst everyone involved.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

​Re: Preventing Duplicate Data REAL WORLD

4D Tech mailing list
I thought I would share this real world experience in regard to data, as it is somewhat humorous, but not for the people in the back seat.

Long long ago, in another country, I was a police member patrolling the downtown streets of a city in economic boom circumstances. There were lots and lots of young males from across the country in the city spending their money on things that too many young men spend their money on. As would be the case many of these young men would find themselves in the backseat of police cars all across the city.

As part of the process we needed to make a decision to release with a piece of paper OR haul down to jail.

Part of that process involved a search on the national criminal data base. We could have ID in front of us (i.e. Driver’s License), or they would just give us a name. The search would be performed based on name and date of birth. With frequent regularity there would be a hit of a warrant out for the person. Some were 50 mile radius (get out of town and don’t come back warrants from 1000 miles away), or we want him back to face justice.

Of course we would ask - why did you lie to us. You have been in trouble, there are warrants for your arrest. The description matches close enough (eye colour, hair). Back then tattoos were not so popular so they were good identifiers. Of course they can be changed - so they would only give more confidence of a right identification.

Off to jail we would head, with our back seat occupant complaining that it is not him.

Check him in past the jail sgt. and away we go. We would find out after his finger prints were taken that he was telling the truth - it was not him.

This would happen with some regularity. In a country of (at that time of only 24 million) I would have this happen about once a month, and there were 1100 police in the city.

So even data with the same name (First + Middle + Last) and same Date of Birth can correctly be different data when dealing with people. The records were not duplicates in looking at the finger print data, but back then (and even now) that is not something they can capture in the cars. Could they - yes but there are laws about taking people’s finger prints.

On a side note:
A few years later I developed a photo ID lineup system that was rejected because it was written in house, and I wrote it on a Macintosh instead of MS-DOS.


Jody Bevan
ARGUS Productions Inc.
Developer
Argus Productions Inc. <https://www.facebook.com/ArgusProductions/>




**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ​Re: Preventing Duplicate Data REAL WORLD

4D Tech mailing list

On Tue, 8 Aug 2017 07:25:00 -0600, Jody Bevan via 4D_Tech wrote:
>
> A few years later I developed a photo ID lineup system that was
> rejected because it was written in house,
Stupid reason

> and I wrote it on a
> Macintosh instead of MS-DOS.
Stupid reason

Ohh... gov't

all makes sense now  :)

---------------
Gas is for washing parts
Alcohol is for drinkin'
Nitromethane is for racing
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Loading...