| Bug #56926 | Installer copies /datadir where it should not | ||
|---|---|---|---|
| Submitted: | 22 Sep 2010 11:44 | Modified: | 10 Feb 2011 15:22 |
| Reporter: | Peter Laursen (Basic Quality Contributor) | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server: Documentation | Severity: | S3 (Non-critical) |
| Version: | 5.5.6 | OS: | Windows (7 64 bit) |
| Assigned to: | Paul DuBois | CPU Architecture: | Any |
| Tags: | qc, regression | ||
[24 Sep 2010 23:23]
Vladislav Vaintroub
Perhaps this needs an explanation. This is not redundant data, this is pristine state data. In the older installers there was no possibility to get clean state after database has been used. Those files in "Program Data" , for service you created during installation are already "dirty", there is already at least the root password you entered during the installation. So, these files in "Program Files" are the clean state, and they could be used if you create a new instance (you would need to manually copy it somewhere though, there is no help from installer for it). The size of the "clean state" data directory under "Program Files" is minimal, I see no problem and no regression here.
[25 Sep 2010 7:40]
Peter Laursen
I never claimed that the disk space is a problem. The main problem is that this suddenly appears with no explanation, what is confusing. It this new behavior between 5.5.5 and 5.5.6 or earlier in 5.5? Where in changelogs/release notes is this listed? I still think it is a regression to use the installation folder (\ProgramFiles) due to UAC-feature of recent Windows. Also if user should start the server not as a service but from command line and forgets the --datadir, as the server will then use the 'pristine' folder. This 'pristine' folder should also be somewhere in 'Program Data' so that it is *only used when specified by user and never automatically by server*. But I really do not see any need for this, as user will be able to get a .zip archive and extract a 'pristine' folder from here.
[25 Sep 2010 7:51]
Peter Laursen
Let me add the question that if it is so important to have a 'pristine' copy, can I then assume that the various RPM installers for Linux does the same? :-) If not (what I assume) then why is it required on only one OS (Windows)? What is particular here?
[25 Sep 2010 10:09]
Vladislav Vaintroub
There is no initial database on any OS, but on Windows. One reason is missing Windows-conform mysql_install_db (Iperl script mysql_install_db.pl is not Windows-conform, since it requires Perl, and besides it is undocumented). Second reason is that ConfigWizard tool requires initial database to create an instance.
[25 Sep 2010 10:26]
Vladislav Vaintroub
As for data directory in ProgramFiles, it is meant to be readonly, so it is UAC-conform. ProgramData is meant to be user data (readwrite). ProgramFiles can store readonly files. And the reason things changed is that .msi (and .zip for that reason) are done differently. Packaging is integrated into the CMake build, rather than done post-build. This is also the reason why zip and msi layout are almost identical, except that extra stuff in ProgramData (which is considered readwrite user data, i.e it is copied here just to please ConfigWizard and never touched again by MSI)
[25 Sep 2010 11:01]
Peter Laursen
Now listen .. this is not a proper reply: "Second reason is that ConfigWizard tool requires initial database to create an instance." .. maybe so but on 5.1 this 'shadow'/'pristine' datadir is not there. Maybe it gets created and next deleted on 5.1 -- I cannot tell. But uyu are not supposed to provide 'maybe's but 'answers' instead. Please provide *reasons* and not *excuses*! The 3 most important points for me are: 1) It is confusing for me and it *wastes my time* that such 'shadow'/'pristine' datadir suddenly shows up without any (new) reason. When I see this I will have to research it, report/ask etc. I am pretty much sure that you will soon get other reports too where users are confused about this. (a little provoking I'd say that your laziness saved you ½ hour but you wasted 100 hours of other people). 2) If you want a 'pristine' datadir (what I personally do not think there is any reason for) then >> there is no reason why there should be on one OS only. If it is relevant, it is relevant for every binary package distributed for any OS. >> such 'pristine' datadir should *absolutely not* be in a position where the server will use it automatically. It should be in a position where it is only used if *explicitly specified by user* (as parameter to 'mysqld'). 3) This new packaging concept does not seem to be documented? Actually I do read release notes before I install a new version. But I could have missed/overlooked it of course.
[25 Sep 2010 11:19]
Peter Laursen
Besides there was no reply to my question if this was introduced between 5.5.5 and 5.5.6 or before.
[25 Sep 2010 11:31]
Peter Laursen
"it is meant to be readonly" .. but it is NOT! See this: C:\Users\Peter>attrib "C:\Program Files\MySQL\MySQL Server 5.5\data\test\.empty" A C:\Program Files\MySQL\MySQL Server 5.5\data\test\.empty And btw: what stupid UNIX mismatch is this "\test\.empty" thing on Windows? I do not care about *what it is meant to be* - only *what it is*. This is probably the worst kind of 'left hand work' I have ever seen in my life!
[25 Sep 2010 11:32]
Vladislav Vaintroub
Peter, I do not know exactly why you waste your time looking into installation layout and filing bugs that are not. It was discussed internally, I'm sorry you were not asked. To be exact pristine state was my idea, and other people did not mind / found this reasonable. Expecting people to download a zip and extract data dir from it when they already have MSI installation is not reasonable, in my opinion. Since this installer is written from scratch you can expect some changes, too.
[25 Sep 2010 11:46]
Peter Laursen
1)
I do not request to be *asked* - but I expect to find changes in packaging concepts *documented*.
Could you please reply:
* where was this packaging concept changed?
* where is it documented? I am pretty sure that a similar change with a RPM installer would be properly documented!
2)
"I do not know exactly why you waste your time looking into
installation layout and filing bugs that are not".
a>> of course I get confused when I see a 'datadir' suddenly occurring where it did not before. I need to understand it because I am supporting 50+ MySQL users (mostly Windows users) every day and I need to know details.
b>> I want to know the reason why folders and files exist on my system (I am 'old-fashioned' in that respect).
(and besides I don't like the 'tone' in your reply, but I realize I may have provoked it myself. So let us cool down! I never claimed this is critical or serious. But I need to know what is happening!).
1) Why is a 'pristine' datadir relevant on Windows when not elsewhere?
2) Why is it in a position so that it will sometimes be used automatically by server and not only when explicitly specified by user?
3) Why does a 'hidden Unix-file' ({installation folder}\data\test\.empty) occur on Windows?
[25 Sep 2010 12:34]
Vladislav Vaintroub
1)
* where was this packaging concept changed?
I do not know, maybe 5.5.5? (I'm not a build team member, I cannot exactly answer that)
* where is it documented? I am pretty sure that a similar change with a RPM installer would be properly documented!
There were big changes in RPM too. Documented in source code I guess.
- Why is a 'pristine' datadir relevant on Windows when not elsewhere?
I tried to explain that already . datadir (whether pristine or not) is not relevant anywhere, but on Windows. It is because MySQL on Windows does not have other documented ways to create an instance (which would be mysql_install_db elsewhere). initial database files are only produced and packaged on Windows (RPMs might have empty data/test , but no system tables).
You can remove this directory if you like, it will not affect your installation.
- Why is it in a position so that it will sometimes be used automatically by server and not only when explicitly specified by user?
Don't understand the "sometimes automatically" . Server is (more or less)predictable, it would not sometimes want to use this datadir and next time other datadir :) If you ask why datadir is in "<installdir>/data" directory - it is because it is here in any other configuration.
3) Why does a 'hidden Unix-file' ({installation folder}\data\test\.empty) occur on Windows?
It is something that affects zip/tar.gz packaging more and could be filtered out in MSI (and I think it is even filtered in ProgramData). It is a CMake bug http://www.cmake.org/Bug/view.php?id=8767. How it affects us is explained here http://bazaar.launchpad.net/~mysql/mysql-server/mysql-5.5/annotate/head%3A/sql/CMakeLists....
In short at least one file has to be INSTALL()ed(in cmake sense of INSTALL), for directory, otherwise directory is not packaged with CPack.
[25 Sep 2010 12:35]
Peter Laursen
Ok .. the .empty file is not a big issue. It does not breadk any functionality: FROM cmd.exe: C:\Program Files\MySQL\MySQL SERVER 5.5\bin>mysqld --NO-defaults -- connect wiht client on port 3306, root user/empty password SHOW DATABASES /* Database ------------------ information_schema mysql performance_schema test */ USE test; SHOW TABLES; -- empty set .. but the file is not *read only*. UAC does not mean that fieles are 'read only - it means that writing to the file is 'virtualized' to user's own folder.
[25 Sep 2010 12:50]
Peter Laursen
I know that I can safely delete this 'pristine' folder. Let me repeat that 5.1 installers/config wizards did not leave such behind. So the reason is definitely not Windows-related. It is internal for MySQL installer/configuration routines. What 5.1 could do/could not do there is no reason that 5.5 cannot/can on the same system. But maybe rather build team should explain? (btw: in Windows-context 'source code' in this context must refer .msi scripts and config wizard code that are not included in the source archive for Windows. Doesn't same apply to RPM's? The C-source (.cc, .h etc. files) have no knowledge of (any kind of) packaging applied after compilation I believe)
[25 Sep 2010 12:54]
Vladislav Vaintroub
C:\Program Files\MySQL\MySQL Server 5.5\bin>mysqld 100925 14:53:13 [Warning] Can't create test file C:\Program Files\MySQL\MySQL Se rver 5.5\data\WIN2008.lower-test 100925 14:53:13 [Warning] Can't create test file C:\Program Files\MySQL\MySQL Se rver 5.5\data\WIN2008.lower-test 100925 14:53:13 [Note] Buffered information: Performance schema disabled (reason : start parameters). 100925 14:53:13 [Note] Plugin 'FEDERATED' is disabled. InnoDB: The InnoDB memory heap is disabled InnoDB: Mutexes and rw_locks use InnoDB's own implementation InnoDB: Compressed tables use zlib 1.2.3 100925 14:53:13 InnoDB: Operating system error number 5 in a file operation. InnoDB: The error means mysqld does not have the access rights to InnoDB: the directory. It may also be you have created a subdirectory InnoDB: of the same name as a data file. InnoDB: File name .\ibdata1 InnoDB: File operation call: 'create'. InnoDB: Cannot continue operation.
[25 Sep 2010 13:11]
Vladislav Vaintroub
Let me explain how (conceptually, it differs in some detail) MSI installer is built now, so you understand that "source" . MSI installer is in source, it is in packaging/WiX. There are couple fo macros that create WiX source, and if you happen to have Wix 3.0, you can do following cmake . devenv mysql.sln /build relwithdebinfo devenv mysql.sln /build relwithdebinfo /project MSI This creates .msi in your <builddir>/packaging/WIX directory. The source wsx file is created by CMake scripts, "compiled" with Wix's candle.exe and "linked" into MSI with Wix's light.exe. This would be ideally done by builtin CPack MSI generator, but since CPack does not have one, it is done by in-house developed one. This is a high-level of how MSI is built. It is possible to add custom stuff to it, and some "blobs" are added to the final installer (ConfigWizard, and banners) is using this extensibility. In the past (5.1) it was built differently, it was a static Wix template in non-public repository, that used ZIP as source. The problem with static templates was that they were not flexible, and they had to change every time something new executable or file had to be added to installation. So this lacked maintainability. Is this ok as explanation?
[25 Sep 2010 13:20]
Peter Laursen
1) I do not understand the next-previous post starting C:\Program Files\MySQL\MySQL Server 5.5\bin>mysqld 100925 14:53:13 [Warning] Can't create test file C:\Program Files\MySQL\MySQL Se rver 5.5\data\WIN2008.lower-test ... why do you post this? Did you discover an issue? 2) previous post "Is this ok as explanation?" .. too technical for me. I have no clue about WiX etc. (I have not been writing a line of code except for SQL, HTML and a little bit of VBA for almost 20 years now). I only want such changes documented and explained for users in a way that does not require knowledge of any systems used to build MySQL software. If users can handle the SQL layer and basic installation/configuration routines don't expect more from users.
[25 Sep 2010 13:30]
Vladislav Vaintroub
1) I do not understand the next-previous post starting C:\Program Files\MySQL\MySQL Server 5.5\bin>mysqld 100925 14:53:13 [Warning] Can't create test file C:\Program Files\MySQL\MySQL Se rver 5.5\data\WIN2008.lower-test ... why do you post this? Did you discover an issue? No. it was a reply to similar post from you, where you were trying to show UAC and virtualized folders, changing to bin directory from the command line and starting mysqld.exe there. And it meant "can't repeat". 2) previous post "Is this ok as explanation?" .. too technical for me. I have no clue about WiX etc. Well, since you started this "source" thing... On non-technical level I'm trying to tell that installer is in source.
[29 Sep 2010 10:25]
Susanne Ebrecht
I will set this to documentation. I think all this could be documented more clear so that users won't get confused.
[10 Feb 2011 15:22]
Paul DuBois
Thank you for your bug report. This issue has been addressed in the documentation. The updated documentation will appear on our website shortly, and will be included in the next release of the relevant products. Added to 5.5.5 changelog, http://dev.mysql.com/doc/refman/5.5/en/news-5-5-5.html: The Windows MSI package creates and sets up the data directory that the installed server will use, but now it also creates a pristine “template” data directory named data under the installation directory. After an installation has been performed using this package, the template data directory can be copied to set up additional MySQL instances. See Section 5.6, “Running Multiple MySQL Instances on One Machine”. The multiple-instance section now has a new subsection describing how to set up new data directories. This section mentions the "template" data directory: http://dev.mysql.com/doc/refman/5.5/en/multiple-data-directories.html And the Windows-layout page now mentions the template directory: http://dev.mysql.com/doc/refman/5.5/en/windows-installation-layout.html
Description: The 5.5.6 installer copies a redundant \data folder to installation folder - This is a regression. Recent 5.1.x installers (correctly) don't. How to repeat: 1) From 'cmd.exe': ******************* C:\Users\Peter>cd \program files\mysql\mysql server 5.5 C:\Program Files\MySQL\MySQL Server 5.5>dir Disken i drev C har ikke noget navn. Diskens serienummer er 40AF-D553 Indhold af C:\Program Files\MySQL\MySQL Server 5.5 21-09-2010 11:56 <DIR> . 21-09-2010 11:56 <DIR> .. 21-09-2010 11:54 <DIR> bin 21-09-2010 11:54 <DIR> data <---- note this one 21-09-2010 11:54 <DIR> include 21-09-2010 11:54 <DIR> lib 13-09-2010 13:01 4.913 my-huge.ini 13-09-2010 13:01 20.658 my-innodb-heavy-4G.ini 13-09-2010 13:01 4.887 my-large.ini 13-09-2010 13:01 4.896 my-medium.ini 13-09-2010 13:01 2.874 my-small.ini 26-08-2010 15:56 13.128 my-template.ini 22-09-2010 13:23 8.944 my.ini 21-09-2010 11:54 <DIR> mysql-test 21-09-2010 11:54 <DIR> scripts 21-09-2010 11:54 <DIR> share 7 fil(er) 60.300 byte 9 mappe(r) 40.783.044.608 byte ledig C:\Program Files\MySQL\MySQL Server 5.5> 2) Inspecting my.ini *********************** The my.ini reads (and this file was created by the configuration wizard) ... #Path to the database root datadir="C:/ProgramData/MySQL/MySQL Server 5.5/Data/" ... - and this is the one that is used by the server(I can tell for sure by simply comparing folders in filesystem with databases and tables in MySQL). I had a /datadir in 'ProgramData' from an 5.5.5 installation that was uninstalled before installing 5.5.6. So I am not able to tell if it would be created correctly at all with the 5.5.6 installation. But if not the server will not start with a configuration built by wizard/default settings. Suggested fix: Make it work like 5.1 with one suggested improvement: Move my.ini to "C:/ProgramData/MySQL/MySQL Server 5.5" for handling UAC correctly in recent Windows.