Bug #86781 Queries don't work with stored functions
Submitted: 21 Jun 2017 17:21 Modified: 22 Jul 2017 14:54
Reporter: Oleksii Vynnychenko Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: Stored Routines Severity:S2 (Serious)
Version:5.7.18 OS:Microsoft Windows
Assigned to: CPU Architecture:Any
Tags: query, SF, stored function

[21 Jun 2017 17:21] Oleksii Vynnychenko
Description:
This is a related issue with the one posted some time ago:
https://bugs.mysql.com/bug.php?id=86220
It's a bit old now and categorized, so created this new one. Don't know how to link issues here.

Anyway... There seems to be a bug where queries not working when using stored functions (SF) in them.

How to repeat:
Here's a SQL listing to recreate DB and then queries with description (sorry, it's a bit complicated but I've simplified it as I could):

CREATE DATABASE bug_mysql;
USE bug_mysql;
# -----------------------------------------------------------------------------------------------
# creating tables
DROP TABLE IF EXISTS account_to_company;
CREATE TABLE account_to_company
(
    account_id INT NOT NULL,
    company_id INT NOT NULL,
    PRIMARY KEY (account_id, company_id)
);
INSERT INTO account_to_company (account_id, company_id) VALUES (1, 1);
INSERT INTO account_to_company (account_id, company_id) VALUES (2, 1);
INSERT INTO account_to_company (account_id, company_id) VALUES (3, 1);
INSERT INTO account_to_company (account_id, company_id) VALUES (4, 1);
INSERT INTO account_to_company (account_id, company_id) VALUES (5, 1);

DROP TABLE IF EXISTS accounts;
CREATE TABLE accounts
(
    id                   INT AUTO_INCREMENT
        PRIMARY KEY,
    username             VARCHAR(500) DEFAULT '' NOT NULL,
    name                 VARCHAR(500)            NULL,
    first_name           VARCHAR(500)            NULL,
    last_name            VARCHAR(500)            NULL
);
INSERT INTO accounts (username, name, first_name, last_name) VALUES ('admin', 'Se', null, null);
INSERT INTO accounts (username, name, first_name, last_name) VALUES ('editor1', 'Sc', null, null);
INSERT INTO accounts (username, name, first_name, last_name) VALUES ('GA', 'Ga', null, null);
INSERT INTO accounts (username, name, first_name, last_name) VALUES ('GE', 'Ga', null, null);
INSERT INTO accounts (username, name, first_name, last_name) VALUES ('s', 'Ol', null, null);
INSERT INTO accounts (username, name, first_name, last_name) VALUES ('GaA', 'Ga', null, null);
INSERT INTO accounts (username, name, first_name, last_name) VALUES ('someeditor', 'Some', null, null);
INSERT INTO accounts (username, name, first_name, last_name) VALUES ('somemanager', 'Some', null, null);

DROP TABLE IF EXISTS article_workflows;
CREATE TABLE article_workflows
(
    id                         INT AUTO_INCREMENT
        PRIMARY KEY,
    name                       VARCHAR(250)           NOT NULL
);
INSERT INTO article_workflows (name) VALUES ('ESS');

DROP TABLE IF EXISTS companies;
CREATE TABLE companies
(
    id                   INT AUTO_INCREMENT
        PRIMARY KEY,
    name                 VARCHAR(250) NOT NULL
);
INSERT INTO companies (name) VALUES ('AA');

DROP TABLE IF EXISTS custom_field_values;
CREATE TABLE custom_field_values
(
    id                   INT AUTO_INCREMENT
        PRIMARY KEY,
    custom_fields_id     INT           NOT NULL,
    owner_id             INT           NOT NULL,
    value                VARCHAR(4000) NULL
);

DROP TABLE IF EXISTS custom_fields;
CREATE TABLE custom_fields
(
    id                            INT AUTO_INCREMENT
        PRIMARY KEY,
    name                          VARCHAR(250)  NOT NULL,
    field_type                    VARCHAR(50)   NOT NULL,
    owner_type                    VARCHAR(50)   NOT NULL,
    article_workflow_id           INT           NULL,
    properties                    VARCHAR(4000) NULL,
    strict_name                   VARCHAR(250)  NULL
);
INSERT INTO custom_fields (name, field_type, owner_type, article_workflow_id, properties, strict_name)
    VALUES ('Is Editor', 'isMemberOfTeam', 'accounts',  1, '{"teamDefinitionIds":[1]}', 'IsEditor');

DROP TABLE IF EXISTS team_definitions;
CREATE TABLE team_definitions
(
    id                            INT AUTO_INCREMENT
        PRIMARY KEY,
    name                          VARCHAR(250)           NOT NULL,
    article_workflow_id           INT DEFAULT '0'        NOT NULL,
    owner_type                    VARCHAR(50)            NOT NULL,
    strict_name                   VARCHAR(250)           NULL
);
INSERT INTO team_definitions (name, article_workflow_id, owner_type, strict_name) VALUES ('Editors', 1, 'companies', 'Editors');

DROP TABLE IF EXISTS team_entries;
CREATE TABLE team_entries
(
    id                   INT AUTO_INCREMENT
        PRIMARY KEY,
    type                 VARCHAR(50)  NOT NULL,
    member_id            INT          NOT NULL,
    team_definition_id   INT          NOT NULL,
    owner_id             INT          NOT NULL
);
INSERT INTO team_entries (type, member_id, team_definition_id, owner_id) VALUES ('account', 2, 1, 1);
INSERT INTO team_entries (type, member_id, team_definition_id, owner_id) VALUES ('account', 4, 1, 1);

# -----------------------------------------------------------------------------------------------
# creating TD view
CREATE OR REPLACE VIEW team_definitions_to_accounts AS
    SELECT
        td2a.id         AS id,
        td2a.account_id AS account_id
    FROM (
        SELECT
            td.id         AS id,
            (
                CASE te.type
                    WHEN 'account' THEN te.member_id
                    WHEN 'exclude_account' THEN te.member_id
                    ELSE NULL
                END
            ) AS account_id,
            if((te.type = 'exclude_account'), 0, 1) AS include_in_result
        FROM team_definitions td
            JOIN team_entries te ON te.team_definition_id = td.id
    ) td2a
    GROUP BY td2a.id, td2a.account_id
    HAVING (bit_and(td2a.include_in_result) <> 0);

# -----------------------------------------------------------------------------------------------
# creating SFs
DROP FUNCTION IF EXISTS IsMemberOfTeam;
CREATE FUNCTION IsMemberOfTeam(accountId INT, teamDefinitionIds JSON, companyId INT)
    RETURNS VARCHAR(4000)
    DETERMINISTIC
    READS SQL DATA
    BEGIN
        IF accountId IS NOT NULL AND json_type(teamDefinitionIds) = 'ARRAY' THEN
            RETURN if(
                exists(
                    SELECT id
                    FROM team_definitions_to_accounts td2a
                        WHERE json_contains(teamDefinitionIds, cast(td2a.id AS CHAR), '$') AND td2a.account_id = accountId
                            AND IF(companyId IS NOT NULL AND companyId != 0 AND td2a.owner_type = 'companies', td2a.owner_id = companyId, TRUE)
                ), 'true', 'false'
            );
        ELSE RETURN 'false';
        END IF;
    END;

DROP FUNCTION IF EXISTS GetCustomFieldValue;
CREATE FUNCTION GetCustomFieldValue(ownerObjectId INT, fieldType VARCHAR(50), serializedValue VARCHAR(4000), propertyObject VARCHAR(4000))
    RETURNS VARCHAR(4000)
    DETERMINISTIC
    READS SQL DATA
    BEGIN
        CASE fieldType
            WHEN 'isMemberOfTeam' THEN
                RETURN if(json_valid(propertyObject), IsMemberOfTeam(ownerObjectId, propertyObject->>'$.teamDefinitionIds', propertyObject->>'$.companyId'), 'false');
            -- any other field type
            ELSE RETURN serializedValue;
        END CASE;
    END;

# -----------------------------------------------------------------------------------------------
# creating NOT bugged view
CREATE OR REPLACE VIEW `team_definitions_to_accounts` AS
SELECT id, owner_type, owner_id, account_id
FROM
    (
        SELECT
            td.id,
            td.owner_type,
            te.owner_id,
            (
                CASE te.type
                    WHEN 'account' THEN te.member_id
                    WHEN 'company_accounts' THEN a2c.account_id
                    WHEN 'exclude_account' THEN te.member_id
                    ELSE NULL
                END
            ) AS account_id,
            IF(te.type = 'exclude_account', 0, 1) AS include_in_result

        FROM team_definitions td
            INNER JOIN team_entries te ON te.team_definition_id = td.id
            LEFT JOIN account_to_company a2c ON te.type = 'company_accounts' AND a2c.company_id = te.member_id
    ) td2a
GROUP BY id, owner_type, owner_id, account_id
HAVING BIT_AND(include_in_result) != 0;

# creating main view
CREATE OR REPLACE VIEW main_view AS
    SELECT
        a.id                                                                 AS object_with_custom_fields_id,
        a.id                                                                 AS account_id,
        a.username,
        cf.id,
        cf.name,
        cf.strict_name,
        cf.field_type,
        cf.owner_type,
        cf.article_workflow_id                                               AS article_workflow_id,
        cf.properties,
        a.id                                                                 AS value_owner_id,
        GetCustomFieldValue(a.id, cf.field_type, cfv.`value`, cf.properties) AS `value`
    FROM
        accounts a
        INNER JOIN custom_fields cf ON cf.owner_type = 'accounts'
        INNER JOIN article_workflows aw ON aw.id = cf.article_workflow_id
        LEFT JOIN custom_field_values cfv ON cf.id = cfv.custom_fields_id AND a.id = cfv.owner_id;

# -----------------------------------------------------------------------------------------------
# reproducing bug

# This is simple query. Works perfect
SELECT object_with_custom_fields_id, acf.`value`->>'$', acf.`value`->>'$'='true'
FROM main_view acf
WHERE strict_name='IsEditor';

# Note the {acf.`value`->>'$'='true'} field in SELECT

# This is the same query but with additional condition {acf.`value`->>'$'='true'} in WHERE that return nothing,
# when it should return 2 entries (you can look at previous query)
SELECT object_with_custom_fields_id, acf.`value`->>'$', acf.`value`->>'$'='true'
FROM main_view acf
WHERE strict_name='IsEditor' And acf.`value`->>'$'='true';

# Both SFs are DETERMINISTIC (and indeed they are) with specified READS SQL DATA.

# THE END
# -----------------------------------------------------------------------------------------------

It would be very nice if somehow you could fix this issue. Makes usage of SFs in VIEWs, queries impossible.

Thank you
[21 Jun 2017 18:27] Sinisa Milivojevic
Hi!

For the starters, I do understand what "not bugged view" means. I do not get what is "TD view". 

Second, I noticed that you are using VARCHAR instead of the JSON data type. Are results better with JSON data type ????

Last, is your intention to unquote the result, so that is why you use ->> instead of -> ???

Thanks a lot in advance !!!!!
[22 Jun 2017 9:16] Oleksii Vynnychenko
Oh well, my mistake, forgot to delete that 'not_bugged view' from listing, it's a copy of that 'TD view'. The latter is just a view that we're using with a bit convoluted at first glance logic, but never had problems with it.

1. About JSON, I'll be able to play with it on Saturday or Sunday, maybe I'll try to simplify it more (already spend huge amount of time to narrow our DB to the one in listing). I'll reply then of course.

2. About ->> and -> . Yes, we need unquoted value, so using that neat '->>' instead of json_unquote(json_extract()). Again, I'll try to play with it on weekend, maybe remove usage of JSON while still getting this issue.

Thank you!
[22 Jun 2017 10:14] Oleksii Vynnychenko
Our main goal is to re-use large chunks of SQL in dozens of places, views, which SFs should solve fine in theory. Maybe you have a suggestion on how to re-use the other way than SFs? Or maybe some special flags that will force to inline all SQL code from SFs into views or queries?
[22 Jun 2017 14:54] Sinisa Milivojevic
Hi ,

We are eagerly awaiting for your feedback with test case that is corrected and changed as we have agreed.
[23 Jul 2017 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".